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The  rapidly  changing  complexity  of  the  Global  War  on  Terrorism  has  changed  the 
approach  to  equipping  forward  deployed  military  forces.  Combatant  Commanders 
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capabilities  and  reduce  risk  for  individual  Soldiers.  To  address  this  challenge  the  U.S. 
Army  established  the  Rapid  Equipping  Force  (REF)  to  assess  emerging  requirements,  to 
propose  solutions  to  those  requirements,  and  to  implement  those  solutions  in  an  expedient 
time  frame.  Unfortunately,  the  REF  lacks  a  consistent  analytical  methodology  for 
developing  alternative  materiel  solutions.  To  address  the  need  for  a  Human  Systems 
Integration  analysis  method,  the  authors  developed  an  Assessment  Based  Rapid 
Acquisition  HSI  Analysis  Tool  (ABRAHAM)  capable  of  generating  tailored  surveys  and 
evaluating  these  surveys  for  unacceptable  risks  to  Soldiers.  To  validate  ABRAHAM’s 
concept  and  content,  ABRAHAM  was  showcased  in  three  Department  of  Defense 
settings,  the  Human  Factors  Engineering  Technical  Advisory  Group,  the  REF,  and  the 
United  States  Marine  Corps’  Operational  Test  and  Evaluation  Activity.  The  ABRAHAM 
appears  to  fill  a  gap  in  the  current  library  of  HSI  tools.  Based  on  the  feedback  provided 
during  the  product  showcases,  there  is  sufficient  interest  and  technological  maturity  to 
further  develop  ABRAHAM  to  serve  both  the  traditional  and  rapid  acquisition  processes. 
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I.  INTRODUCTION 


A.  OVERVIEW 

The  rapidly  changing  demands  and  increasing  complexity  of  the  Global  War  on 
Terrorism  have  necessitated  an  improved  approach  for  equipping  forward  deployed 
military  forces.  Combatant  Commanders  conducting  operations  now  require  rapid  and 
functional  materiel  solutions  to  enhance  mission  capabilities  and  ultimately  mitigate  the 
risks  to  the  individual  Soldier.  Previous  methods  for  obtaining  materiel  solutions  are 
limited  to  the  capacity  of  the  lengthy  traditional  acquisition  process.  This  process 
requires  solicitation  and  evaluation  of  proposals,  documentation  of  contract  bidding,  and 
once  a  contractor  is  selected,  intense  management  of  the  contractor  to  ensure  compliance 
with  all  specifications  of  the  contract.  To  counter  this  problem,  the  U.S.  Army 
established  an  agency  devoted  to  meeting  the  requirements  of  the  Combatant 
Commanders.  This  agency,  known  as  the  Rapid  Equipping  Force  (REF),  was  given  three 
goals:  a)  to  assess  emerging  requirements  for  materiel  solutions;  b)  to  quickly  match 
those  solutions  to  the  emerging  operational  requirements  of  combatant  commanders;  and, 
c)  to  suggest  solutions  that  can  be  implemented  in  an  operationally  useful  time  frame. 

B.  THE  RAPID  EQUIPPING  FORCE  (REF) 

The  concept  of  the  REF  first  arose  in  a  meeting  between  U.S.  Army  Colonel 
Bruce  Jette  and  General  John  Keane  in  2002.  At  the  time,  U.S.  efforts  were  concentrated 
on  the  search  for  Osama  Bin  Laden  and  military  intelligence  directed  specific  attention  to 
the  caves  of  Afghanistan.  Colonel  Jette  and  General  Keane  met  to  discuss  the  various 
options  available  to  efficient  and  effective  cave  searches.  Previously,  Army  doctrine 
employed  Soldiers  to  perform  the  tasks.  From  the  two  officers’  perspectives,  this  method 
seemed  to  be  akin  to  the  tactics  used  almost  four  decades  ago  in  Vietnam.  The  discussion 
led  to  an  innovative  solution  that  reduced  manpower  and  significantly  mitigated  the  risks 
to  Soldiers.  The  officers  developed  the  concept  of  leveraging  robotic  capabilities  to 
conduct  the  search.  Within  a  time,  Colonel  Jette  acquired  a  remote-controlled  robot 
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using  commercial  off  the  shelf  technology  (COTS)  and  took  it  to  Afghanistan  where  it 
was  received  with  good  success.  This  incident  began  the  evolution  of  the  Rapid 
Equipping  Force. 

The  concept  of  REF  grew  in  popularity.  The  mission  of  the  REF  soon  expanded 
its  range  of  available  solutions  beyond  robotic  search  drones.  Rapid  equipping  included 
any  solution  that  optimized  force  protection,  survivability,  or  lethality.  The  REF’s  goal 
became  to  equip  operational  commanders  with  rapidly  employable  solutions  of 
commercial  off  the  shelf  (COTS),  government  off  the  shelf  (GOTS)  technologies,  or 
future  force  technologies  that  would  optimize  unit  perfonnance.  In  return,  operational 
commanders  would  inform  the  Army  leadership  the  extent  to  which  the  materiel  solutions 
would  allow  them  to  keep  abreast  of  an  adaptive  enemy. 

Eventually,  objectives  of  the  of  the  REF  were  instantiated  as:  equip  the  force, 
insert  COTS,  GOTS,  or  future  force  technologies  into  the  force,  and  assess  product 
inserted. 

The  first  objective,  equipping  the  force,  focuses  on  providing  materiel  solutions  to 
the  current  force  in  order  to  rapidly  eliminate  operational  capability  shortfalls.  The 
second  objective,  inserting  technologies  into  the  force,  provides  a  conduit  for  future  force 
technologies,  threshold  capabilities,  or  surrogates  into  the  current  operational 
environment  for  assessment  by  committed  forces.  The  last  objective,  assessing  inserted 
products,  provides  evaluations  and  lessons  learned  back  to  Army  leadership  for  the 
purpose  of  refining,  modifying,  and  streamlining  Army  acquisition  decisions  in  support 
of  the  force.  Together,  these  objectives  create  an  automatic  feedback  loop  to  allow 
continual  learning  and  development  of  best  practices,  and  a  strategy  that  supports  the 
complexity  of  the  Global  War  on  Terrorism. 

C.  THE  REF  PRIORITY 

As  stated  in  the  REF  mission  (www.ref.army.mil),  the  agency  works  directly  with 
operational  commanders  to  find  promising  materiel  solutions  for  identified  operational 
requirements.  Consequently,  one  of  the  REF’s  key  tasks  is  to  frequently  survey 
battlefield  operations  for  possible  equipment  improvements.  As  problems  or  gaps  are 
identified  by  forward  deployed  REF  personnel,  a  request  is  forwarded  to  REF 
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headquarters  for  analysis  and  solution  alternatives  are  generated.  Next  the  Director 
selects  a  final  product  to  field.  The  REF  completes  the  process  in  less  than  180  days, 
shorter  than  the  time  required  in  the  typical  acquisition  cycle. 

D.  THE  TYPICAL  ACQUISITION  CYCLE 

In  July  2004,  Department  of  Defense  (DoD)  5000  instruction  was  published  to 
improve  military  acquisition  by  refining  the  acquisition  lifecycle  process  (see  Figure  1 
below).  The  5000  instruction  describes  a  management  framework  that  is  intended  to 
translate  mission  needs  and  requirements  into  systems  acquisition  programs.  To 
accomplish  this,  the  framework  specifies  the  following  five  phases:  Concept  Refinement, 
Technology  Development,  System  Development  and  Demonstration,  Production  and 
Deployment,  and  Operations  and  Support. 
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Figure  1.  Acquisition  Lifecycle  (REF  Presentation,  COL  Lovett,  2006) 

Each  phase  has  a  distinct  purpose.  In  the  Concept  Refinement  phase,  the  initial 
concept  is  refined  in  order  to  produce  a  strategy  for  acquiring  a  particular  system 
capability.  At  the  conclusion  of  the  Concept  Refinement  phase,  the  Defense  Acquisition 
Board  meets  to  determine  if  the  project  should  advance  to  the  Technology  Department 
phase.  This  transition  from  Concept  Refinement  to  Technology  Development 
encompasses  the  Milestone  A  decision,  which  is  discussed  in  more  detail  below.  The 
Technology  Development  phase,  focuses  on  providing  the  appropriate  set  of  technologies 
to  be  integrated  into  the  system.  This  is  conducted  by  iteratively  assessing  various 
technologies  while  simultaneously  refining  the  user  requirements.  In  order  to  make  a 
decision  whether  or  not  to  proceed  to  the  Development  and  Demonstration  phase, 
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selected  technologies  are  demonstrated  in  a  relevant  environment.  This  phase  includes 
the  Milestone  B  decision.  The  purpose  of  the  System  Development  and  Demonstration 
phase  is  to  demonstrate  that  the  system/system  increment  can  effectively  function  in  its 
target  environment.  Consequently,  a  decision  is  made  regarding  movement  to  Production 
and  Deployment.  This  phase  comprises  the  Milestone  C. 

The  remaining  two  phases  do  not  have  Milestone  decisions  associated  with  them. 
In  the  Production  and  Deployment  phase  independent  operational  tests  and  evaluations 
verify  the  operational  capability  needed  for  satisfying  mission  needs.  In  addition,  the 
system  is  implemented  at  all  locations.  During  the  Operations  and  Support  phase,  a 
program  is  implemented  to  meet  operational  support  requirements.  Efforts  are  focused 
on  sustaining  the  system  over  its  total  lifecycle  in  a  cost-effective  manner. 

According  to  the  framework,  an  acquisition  program  may  begin  at  Milestone  A, 
B,  or  C.  Progression  through  the  program  depends  on  whether  the  system  meets  specific 
criteria  to  continue  to  the  next  acquisition  phase.  Although  the  framework  permits 
programs  to  be  managed  as  a  single  project,  DoD  Instruction  5000.2  states  that  the 
department  prefers  an  evolutionary  acquisition  strategy  that  delivers  a  mature  product  in 
increments.  Under  such  a  strategy,  each  increment  is  to  begin  with  a  Milestone  B 
decision,  and  the  Production  and  Deployment  phase  of  each  increment  is  to  begin  with  a 
Milestone  C  decision.  Figure  1  provides  a  simplified  diagram  of  the  DoD’s  acquisition 
management  framework.  This  refined  process  was  developed  to  encourage  efficiency  and 
flexibility,  without  minimizing  creativity  and  innovation,  when  providing  modem 
technology  to  the  warfighter. 

Despite  this  streamlined  process,  Combatant  Commanders  still  consider  it  to  be 
time-intensive,  cumbersome,  and  generally  inadequate  for  mission  effectiveness.  For 
instance,  U.S.  Army  Major  General  Roger  Nadeau,  commanding  officer  of  the  Armament 
Research,  Development,  and  Engineering  Center  (ARDEC),  stated  “traditional 
acquisition  isn’t  working”  (Onley,  2005).  Timeliness  of  the  process  plays  is  essential.  If 
the  process  produces  a  good  result  but  it  requires  a  significant  amount  of  time,  it  could  be 
inadequate  for  meeting  mission  needs. 


4 


To  insure  the  fielding  process  is  responsive  and  provides  viable  solutions  to 
address  prevailing  mission  needs,  the  REF  seeks  to  place  new  systems  within  90  to  180 
days.  With  their  focus  clearly  on  immediately  meeting  the  Combatant  Commander’s 
needs  the  REF  explores  existing  and  near-term  technologies  to  provide  solutions  within 
the  acquisition  lifecycle  with  greater  timeliness.  The  REF  also  rejected  the  traditional 
approach  of  solving  every  problem  to  one  that  meets  at  least  51%  of  the  warfighters’ 
challenges  and  problems.  This  important  distinction  results  in  an  environment  in  which 
solutions  are  identified  quickly  and  provided  expeditiously  to  the  warfighter.  Ultimately, 
this  acquisition  approach  embodies  a  new  model  that  adjusts  the  focus  such  that  direct 
involvement  with  the  warlighting  commander,  engagement  with  Soldiers,  and 
observations  and  participation  in  operations  occur  regularly.  Finally,  this  novel  approach 
allows  modifications  to  be  conducted  on  the  systems  as  required.  Together,  these  factors 
create  an  acquisition  process  that  is  more  responsive  to  the  needs  of  Combatant 
Commanders. 
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Figure  2.  REF  Acquisition  Lifecycle  (REF  Presentation,  COL  Lovett,  2006) 

Although  the  REF  acquisition  lifecycle  greatly  shortens  the  timeline  for  procuring 
new  systems,  there  are  tasks  within  the  process  that  can  be  closely  related  to  the 
traditional  acquisition  lifecycle.  Within  the  first  72  hours  of  the  REF  acquisition 
lifecycle,  analyst  must  determine  if  the  request  from  the  field  is  a  valid  requirement  by 
comparing  it  to  the  TRADOC  Capability  Gap  and  Sub  Gap  List.  If  the  request  is 
considered  a  valid  REF  requirement  then  the  REF  analyst  begins  to  refine  the 
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requirement  by  developing  an  initial  Concept  of  Operations  to  address  the  issue.  The 
final  step  of  the  72-hour  process  is  a  decision  brief  given  to  the  REF  Director  to 
determine  if  the  REF  will  pursue  the  project.  This  72-hour  process  is  akin  to  the  Pre- 
Milestone  A  events  including  the  Milestone  A  decision  brief  of  the  Traditional 
Acquisition  Lifecycle. 

With  its  focus  on  NDI  equipment,  the  REF  will  leverage  the  work  of  equipment 
manufactures  in  order  to  skip  much  of  the  work  done  in  the  Technology  Development 
and  System  Integration  phases  immediately  preceding  and  following  the  Milestone  B 
decision  brief  of  the  Traditional  Acquisition  Lifecycle.  The  REF  process  resumes  with 
limited  work  in  what  should  be  the  System  Demonstration  phase  of  the  Traditional 
Acquisition  Lifecycle,  before  concluding  in  a  fielding  action  that  is  most  similar  the  post- 
Milestone  C  Limited  Rate  Production  phase. 

Clearly,  the  abbreviated  REF  lifecycle  cannot  take  full  advantage  of  many  of  the 
checks,  balances,  and  evaluation  methods  of  the  Traditional  Acquisition  Lifecycle.  One 
such  evaluation  method  that  is  not  explicitly  included  in  the  REF  Lifecycle  is  analysis  of 
how  the  Soldier  will  fully  interact  and  perform  with  the  new  system.  A  rapid  analysis  of 
Human  Systems  Integration  issues  will  help  to  ensure  that  a  human-centered  approach 
remains  at  the  forefront  of  any  decision  to  field  new  systems. 

E.  HUMAN  SYSTEMS  INTEGRATION 

Human  Systems  Integration  (HSI)  is  a  relatively  new  discipline  which 
incorporates  the  seven  domains  of  manpower,  personnel,  training,  human  factors 
engineering,  environment,  safety  and  occupational  health,  soldier  survivability,  and 
habitability  (Department  of  Defense,  2003).  Although  the  Human  Systems  Integration 
title  is  new,  the  MANPRINT  Program  (upon  which  HSI  is  based)  has  been  in  existence 
within  the  U.S.  Army  since  the  early  1980’s  (www.manprint.armv.mil/mp-history.asp  ) 

The  HSI  concept  began  during  World  War  II  with  the  work  of  American  and  British 
engineering  psychologists.  The  concept  was  refined  by  the  United  States  Army,  which 
created  the  Manpower  and  Personnel  Integration  (MANPRINT)  management  and 
technical  program  (Booher,  2003  p.xiii).  MANPRINT  originally  consisted  of  six 
domains:  Manpower,  Personnel,  Training,  Human  Factors  Engineering,  System  Safety, 
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and  Health  Hazards.  The  Army  added  Soldier  Survivability  to  the  list  of  domains 
following  fratricide  incidents  during  Operation  DESERT  STORM  (Kleiner  and  Booher, 
2003  p.3).  HSI  advocates  an  approach  to  acquisition  that  places  consideration  of  human 
capabilities  and  limitations  at  the  forefront  of  decisions  regarding  the  design, 
development,  performance,  procurement  and  lifecycle  costs  of  new  systems.  HSI  is 
defined  as  a  “primarily  technical  and  managerial  concept,  with  specific  emphasis  on 
methods  and  technologies  that  can  be  utilized  to  apply  the  HSI  concept  to  systems 
integration”  (Kleiner  and  Booher,  2003). 

HSI  has  grown  from  a  Department  of  Defense  centric  concept  to  an  acquisition 
management  philosophy  used  by  NASA,  the  FAA,  the  Department  of  Transportation  and 
Defense  Contractors  within  the  United  States  and  Canada,  the  Netherlands  and  numerous 
other  countries  worldwide  (Kleiner  and  Booher,  2003).  Other  organizations  have 
expanded  the  list  of  HSI  domains  to  include  Habitability  and  Medical  Factors.  For  the 
purposes  of  this  thesis,  HSI  consists  of  seven  domains  as  defined  by  DoD  Instruction 
5000.2.  These  seven  domains  are  Manpower,  Personnel,  Training,  Human  Factors 
Engineering,  Habitability,  Survivability,  and  Environment,  Safety  and  Occupational 
Health  (which  generally  includes  Medical  Factors)  (Department  of  Defense,  2003). 
While  these  seven  domains  have  generally  existed  as  stovepipes  in  each  branch  of  the 
military,  the  unique  challenge  of  HSI  is  to  coordinate  the  varied  requirements  in  each  of 
the  domains  into  a  coherent  policy  that  maximizes  total  system  performance.  It  should  be 
noted  that  the  integration  of  domains  must  be  a  robust  and  dynamic  endeavor  because  the 
domains  will  not  be  of  equal  importance  in  every  program  (Office  of  the  Assistant 
Secretary  Research  Development  and  Acquisition,  1998). 

Historically,  military  systems  have  often  ignored  or  minimized  the  attributes  of 
the  human  user  while  maximizing  perceived  technological  and  engineering  opportunities 
to  improve  overall  system  performance.  Many  personnel  involved  in  the  research,  design 
and  acquisition  of  military  systems  consider  the  human  element  of  the  system  to  be  the 
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most  adaptable.  Factors  related  to  the  user  were  typically  given  a  low  priority.  As  a 
result,  the  human  was  forced  to  make  unreasonable  accommodations  in  order  to  make  a 
new  system  work  effectively. 

Reports  from  the  Washington  Post  and  posted  on  Center  for  Army  Lessons 
Learned  indicate  that  the  current  design  implementations  of  the  $  1 1  billion  Stryker  results 
in  unacceptable  operating  conditions  for  the  crew.  For  instance,  Soldiers  are  unable  to 
attach  seatbelts  over  their  body  armor  and  are  forced  to  check  tire  pressure  at  least  three 
times  a  day  to  maintain  the  proper  pressure  levels  due  to  the  weight  of  additional  annor 
added  to  the  vehicle  but  not  accounted  for  in  the  design.  Furthermore,  the  on  board 
computer  has  a  tendency  to  overheat  and  shut-down  at  critical  moments  (Smith,  2005; 
Zagaroli,  2005).  Perhaps  the  most  troublesome  flaw  of  the  Stryker  is  the  vehicle’s  horn. 
The  horn  is  not  loud  enough;  therefore,  an  Army  Lessons  Learned  report  recommended 
Soldiers  replace  the  hom  with  a  local  aftennarket  horn  once  in  theater.  To  help  control 
traffic  while  in  motion,  Soldiers  were  directed  to  throw  rocks  and  use  hand  signals, 
thereby  exposing  themselves  to  attack  (Center  for  Army  Lessons  Learned,  2004). 

In  an  effort  to  ensure  that  the  human  element  receives  equivalent  priority  in 
systems  design  and  acquisition,  the  Department  of  Defense  (DoD)  now  requires  program 
managers  to  apply  human  systems  integration  techniques  “to  optimize  total  system 
performance,  operational  effectiveness,  suitability,  survivability,  safety,  and 
affordability”  (Department  of  Defense,  2003).  Enclosure  7  of  DoD  Instruction  5000.2 
requires  the  Program  Manager  to  develop  a  comprehensive  HSI  plan  and  includes 
overarching  guidance  on  items  to  be  addressed  in  each  of  the  seven  domains  (Department 
of  Defense,  2003).  The  Defense  Acquisitions  Guidebook  (Chapter  6)  serves  as  an 
excellent  resource,  providing  further  definition  of  the  HSI  domains,  an  overview  of  HSI 
throughout  the  acquisitions  process,  and  reference  for  basic  military  instructions, 
handbooks,  and  standards  relating  to  HSI  (Department  of  Defense,  2004). 

To  this  end,  numerous  organizations  have  developed  guidance  to  facilitate  the 
integration  of  requirements  from  the  seven  domains.  The  U.S.  Naval  Sea  Systems 
Command  (NAVSEA)  has  developed  a  2-volume  HSI  guide  for  Program  Managers 
involved  in  Navy  ship  systems  (Naval  Sea  Systems  Command,  2003).  NAVSEA  has 
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also  published  HSI  policy  for  acquisition  and  modernization.  This  document  delineates 
the  various  duties  of  organizations  and  establishes  the  idea  of  Program  HSI  Criteria, 
which  are  to  be  used  to  “assess  the  adequacy  of  design  concepts  from  an  HSI 
perspective”  (Naval  Sea  Systems  Command,  2005).  Although  geared  to  previous 
versions  of  the  DoD  5000  series  instructions,  the  U.S.  Air  Force  HSI  Office  created  both 
an  HSI  Requirement  Resource  Guide  and  an  HSI  Domain  Checklist.  These  documents 
were  developed  to  provide  a  structured  walk  through  of  the  HSI  domains  and  to  facilitate 
in  the  generation  of  the  former  Mission  Need  Statement  and  Operational  Requirements 
Document  (Air  Force  Human  Systems  Integration  Office,  2003;  Air  Force  Human 
Systems  Integration  Office,  2003). 

An  interesting  approach  to  HSI  evaluation  is  suggested  by  Dudley-Rawley  and 
Bishop  (2002).  They  offer  a  breakdown  of  HSI  issues  into  four  phases:  The  Taxonomic 
Phase  (HSI  issue  identification),  the  Quantitative  Phase  (HSI  metric  identification  and 
data  collection),  the  Empirical  Phase  (expert  identification  and  analysis),  and  the  Final 
Phase  (database  integration  and  report  generation).  Originally  forwarded  as  an  approach 
for  extended  space  missions,  this  approach  could  be  used  to  build  a  process  by  which  HSI 
analysis  and  evaluations  are  thoroughly  explored  and  stored  as  a  basis  for  analysis  of 
future  systems. 

A  prevailing  method  of  HSI  evaluation  appears  to  communicate  HSI  requirements 
in  terms  of  the  functional  and  mission  requirements  for  the  system  to  be  procured.  By 
conducting  these  top-down  analyses,  evaluators  can  fully  understand  the  role  of  the 
human  in  the  system  and  make  informed  decisions  about  the  trade-offs  between  the  HSI 
domains  and  other  engineering  or  mission  requirements  (Malone  and  Miller,  2003).  The 
HSI  Plan  for  the  Transportation  Coordinator's  Automated  Information  for  Movement 
System,  Version  II  (TC-AIMS  II)  appears  to  follow  this  approach  (TC-AIMS  II  Project 
Management  Office,  2003). 

The  evaluation  methodology  used  to  examine  the  HSI  domain  must  be  intuitive, 
and  aligned  with  the  REF’s  rapid  acquisition  philosophy.  Time  intensive  evaluation 
techniques  such  as  task  analysis  or  requirements  analysis  would  place  a  large  burden  on 
the  REF  analysts  unfamiliar  with  these  techniques.  Because  the  evaluations  methodology 
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should  maximize  the  use  of  research  conducted  by  the  product  manufacturer,  most  of  the 
REF  acquisitions  will  be  through  Non-Developmental  Items  (NDI).  A  non- 
developmental  item  is  defined  as  any  item  that  is  available  commercially  or  any  item 
already  developed  for  use  by  another  local,  state  or  federal  government  organization. 
Non-developmental  items  may  require  some  modification  prior  to  military  use.  The 
Army  has  three  classifications  for  NDI  equipment:  (1)  no  modification  required,  (2) 
ruggedization  required,  (3)  militarization  required  (Steves  1997).  If  properly  managed, 
using  non-developmental  items  can  lead  to  lower  lifecycle  costs,  improved  reliability, 
and  faster  delivery  of  useful  products  to  operational  units. 

DoD  5000.2  directs  that  when  procuring  NDI  products,  care  must  be  taken  to 
ensure  the  new  system  evolves  with  Doctrine,  Organization,  Training,  Materiel, 
Leadership  and  Education,  Personnel  and  Facilities  (DOTMLPF)  changes.  DOTMLPF 
changes  involve  non-materiel  solutions  to  capability  gaps.  For  example  a  change  of 
Doctrine,  such  as  adding  a  series  of  90-degree  turns  to  the  entrance  of  an  installation  can 
be  as  effective  as  spikes  stripes  or  other  materiel  solutions  when  attempting  to  slow  a 
vehicle.  HSI  domain  concerns  can  be  addressed  as  part  of  the  DOTMLPF  analysis  early 
in  the  acquisition  process,  including  the  Functional  Solutions  Analysis  (HSI  VIRTUAL 
SYSCOM  WORKING  GROUP,  2005). 

HSI  practitioners  are  often  challenged  by  NDI  procurements  because  there  is  little 
ability  to  change  system  designs.  Fortunately,  the  Defense  Standardization  Program  has 
developed  guidance  for  exploring  the  operational  requirement,  conducting  market 
research,  identifying  alternatives,  evaluating  the  alternatives,  and  selecting  the  best 
alternative  among  NDI  (Defense  Standardization  Program,  1996). 

Robust  market  research  techniques  should  be  employed  to  identify  the  full  range 
of  products  available  in  commercial  or  other  governmental  environments,  and  an  HSI 
analysis  should  identify  issues,  risks,  and  concerns  that  should  influence  the  decision 
maker’s  determination  of  the  most  appropriate  alternative. 
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F.  THE  INITIAL  NEED 

As  with  all  new  process  models,  opportunities  exist  for  improvement  of  the  tasks 
and  the  process  that  make  up  the  REF  acquisition  lifecycle.  There  is  a  concern  that  each 
REF  analyst  has  developed  his  or  her  own  different  analysis  and  selection  techniques, 
because  a  standardized  analysis  methodology  does  not  exist. 

In  response  to  this  potential  weakness,  the  REF  requested  that  TRADOC  Analysis 
Center  (TRAC)  Monterey  develop  a  standard  methodology  for  REF  analysts.  A  general 
standardized  approach  to  analysis  should  ideally  consider  material  standards,  cost  benefit 
analysis,  and  Soldier  usability.  The  requirements  focused  on  the  minimization  of  time 
required,  without  reducing  its  intuitive  and  flexible  nature.  The  methodology  is  to  be 
divided  into  two  phases  with  time  limits.  Phase  One  must  be  a  quick  analytic  process 
that  provides  initial  recommendations.  This  analysis  is  to  be  conducted  within  72  hours 
of  the  problem  being  submitted.  Phase  Two  is  the  development  of  a  more  detailed  and 
deliberate  analysis  process  to  refine  the  initial  Phase  One  recommendations.  This  is 
conducted  up  to  six  months  after  the  problem  statement.  Once  the  product  has  been 
fielded,  Phase  Three  provides  Soldier  feedback  to  ensure  the  viability  of  the  product. 

In  collaboration  with  TRAC  Monterey,  the  current  thesis  research  uses  the  field  of 
Human  Systems  Integration  (HSI)  to  supplement  a  standard  methodology.  TRAC- 
Monterey  is  developing  an  overall  systems  approach  to  support  the  REF.  Other  research 
supporting  the  TRAC  Monterey  effort  include  implementing  a  Rapid  Ordering  System, 
an  Infonnation  Technology  Management  tool,  and  an  Analysis  of  Alternatives  Decision 
Support  Tool. 
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II.  METHODOLOGY 


A.  GATHERING  ADDITIONAL  INFORMATION 

The  Methodology  chapter  describes  the  procedure  used  by  the  authors  to  gather 
information,  refine  the  Initial  Need  Statement,  and  create  a  solution  to  address  the  REF’s 
need.  This  chapter  is  subdivided  into  the  following  sections:  Gathering  Additional 
Information,  Refining  the  Initial  Need,  Addressing  the  Effective  Need,  and  Validating  the 
Solution.  In  the  chapters  that  follow,  the  authors  present  the  literature  reviews  used  as 
the  basis  for  the  work  on  this  project,  an  instruction  manual  documenting  the  proper 
operation  of  the  software  solution,  and  a  discussion  of  conceptual  and  design  concerns 
and  user  feedback  generated  during  product  demonstration. 

Based  on  the  Initial  Need  Statement  provided  at  the  end  of  the  Introduction,  the 
authors  traveled  to  the  Rapid  Equipping  Force  Headquarters  in  Fort  Belvoir,  Virginia  on 
18-20  July  2006  to  interview  REF  personnel,  gather  additional  REF  process 
documentation,  and  observe  REF  personnel  in  action.  During  this  trip,  the  authors 
interviewed  Colonel  Gregory  Tubbs,  Director,  Rapid  Equipping  Force;  Colonel  Robert 
Lovett,  REF  Project  Manager;  Mr.  John  Geddes,  REF  Technology  Management  Director; 
Mr.  Joe  Rozmeski,  Deputy,  Operations;  Mr.  Scott  Torgerson,  Capabilities  and 
Assessments  Team  Lead;  and  various  other  REF  analysts  working  in  the  Operations 
Department.  Specific  feedback  from  these  meetings  is  provided  in  the  following  four 
paragraphs. 

The  REF  started  with  12  people  on  179  day  temporary  duty  orders  in  2003.  In 
July  2006,  the  REF  organization  was  manned  with  over  100  people,  with  roughly  47 
people  assigned  to  the  Operations  Department.  Military  members  are  now  assigned  for 
three  years  on  Permanent  Change  of  Station  orders,  and  normally  spend  their  first  year  in 
theater  to  gain  an  understanding  of  the  conditions  in  the  forward  operating  areas  (J. 
Rozmeski,  personal  communication,  July  18,  2006). 

As  of  July  2006,  the  REF  was  in  its  seventh  iteration  of  process  refinement  since 
January  2005.  Although  the  common  core  of  the  organization  remains,  the  REF  has 
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undergone  an  organizational  change  approximately  every  three  months.  REF  leadership 
expects  this  fluidity  to  continue  for  the  foreseeable  future  (J.  Rozmeski,  personal 
communication,  July  18,  2006). 

The  REF  has  operated  with  a  focus  on  operations  in  Iraq  and  Afghanistan,  but  the 
REF  mission  is  to  support  the  entire  Army.  As  such,  the  REF  has  established  a  forward 
detachment  in  Djibouti,  Africa  to  support  CJTF-HOA  and  stands  prepared  to  establish 
similar  detachments  where  ongoing  operations  require  a  REF  presence  (J.  Rozmeski, 
personal  communication,  July  18,  2006).  Despite  the  growth  in  manpower  and 
sustainment  of  corporate  knowledge  (due  to  less  frequent  turnover),  the  REF’s  expanded 
mission  requires  clear  operating  processes  and  innovative  analysis  methods  to  meet  the 
timeline  established  by  the  180  REF  Acquisition  Lifecycle. 

According  to  COL  Tubbs,  the  REF  culture  is  driven  by  delivery  of  products,  but 
the  organization  is  process  adverse.  To  improve  its  process,  the  REF  must  do  away  with 
non-value  added  tasks.  Unfortunately,  there  are  few  quantitatively  oriented  analysts  who 
are  skilled  in  quantitative  techniques  at  the  REF;  the  organization  does  not  currently  use 
quantitative  tools  that  can  provide  data  in  support  of  Lean  and  Six  Sigma  process 
improvement  initiatives  (G.  N.  Tubbs,  personal  communication,  July  19,  2006).  For 
example,  the  System  Assessment  Form,  (Appendix  B),  is  too  generic  to  accurately  assess 
the  wide  range  of  equipment  for  which  the  REF  is  responsible  and  offers  little 
opportunity  for  meaningful  statistical  analysis  (S.  Torgerson,  personal  communication, 
July  19,  2006). 

B.  REFINING  THE  INITIAL  NEED  STATEMENT 

With  this  interview  data  and  the  additional  process  documents  obtained  during  the 
trip  to  Fort  Belvoir,  the  authors  identified  two  areas  in  which  the  REF  benefit  from  an 
HSI  evaluation.  First,  inserting  an  HSI  evaluation  early  in  the  development  of  materiel 
solutions  and  non-materiel  solutions  would  ensure  that  REF  analysts  consider  a  human- 
centered  approach  in  their  alternatives  analysis.  Secondly,  inserting  an  HSI  evaluation 
into  product  evaluation  questionnaires  sent  to  Soldiers  in  the  field  would  result  in 
meaningful  feedback  that  could  be  used  to  support  the  decision  to  kill  or  refine  a  system. 
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In  order  to  take  advantage  of  these  opportunities,  the  authors  developed  a  method 
to  conduct  these  evaluations  with  little  or  no  training  in  the  HSI  or  MANPRINT.  To  gain 
a  better  understanding  of  the  concepts  to  be  evaluated  in  these  REF  HSI  evaluations,  the 
authors  reviewed  scientific  and  scholarly  literature  related  to  the  domains  of  HSI.  This 
literature  review,  which  is  included  as  Chapters  III-IX,  revealed  that  HSI  analysis 
techniques  used  in  traditional  acquisitions  environments  are  often  time  intensive 
processes  requiring  expert  knowledge. 

With  a  firm  understanding  of  REF  requirements  and  the  general  tasks  involved  in 
the  analysis  of  each  HSI  domain,  the  authors  refined  the  Initial  Need  Statement  to  create 
a  more  useful  Effective  Need  Statement.  The  new  Effective  Need  Statement  follows: 
The  REF  needs  to  improve  its  analysis  methodology  by  utilizing  both  qualitative  and 
quantitative  tools  and  methods  to  generate  data  for  use  in  alternatives  analysis,  product 
evaluation  and  process  improvement  initiatives.  REF  analysts  must  be  capable  of  using 
these  tools  and  methods  to  meet  time  constraints  imposed  by  the  180-day  REF 
Acquisition  Lifecycle  without  the  need  for  extensive  training  or  expert  knowledge  in  the 
area  of  HSI. 

C.  ADDRESSING  THE  EFFECTIVE  NEED 

To  address  this  Effective  Need  Statement,  the  authors  developed  a  method  of 
automatically  generating  and  analyzing  customized  questionnaires  that  are  tailored  based 
on  the  REF  analyst’s  description  of  the  piece  of  equipment.  The  questions  used  to 
generate  these  surveys  were  derived  from  information  gathered  from  the  literature 
review,  Army  questionnaires,  other  domain  related  checklists,  and  Subject  Matter  Expert 
(SME)  interviews.  All  questions  included  in  the  database  were  reviewed  for  their 
relevancy  to  REF  systems;  those  questions  that  were  deemed  irrelevant  were  modified  or 
excluded  from  the  tool. 

To  display  these  questions  in  an  intuitive  manner,  the  authors  created  an 
application  using  Microsoft  Access  database  software,  hereafter  called  the  Assessment 
Based  Rapid  Acquisition  HSI  Analysis  Module  (ABRAHAM).  Microsoft  Access  is  a 
part  of  the  Microsoft  Office  Professional  software  package,  and  as  such,  the  software  is 
common  to  most  military  and  DoD  computer  systems.  Additionally,  software  developers 
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can  utilize  Microsoft  Access  to  repackage  the  database  in  a  run  time  environment  that 
users  can  employ  without  installing  Microsoft  Access. 

Programmers  can  take  advantage  of  Access’  limited  multi-user  environment  to 
support  simultaneous  work  by  up  to  255  users.  Microsoft  designed  Access  to  easily 
integrate  with  other  Microsoft  Office  products,  including  Microsoft  Frontpage  web 
development  software.  This  close  integration  provides  a  straightforward  method  for 
distributing  HSI  questionnaires  via  the  Internet. 

ABRAHAM  is  designed  to  walk  REF  analysts  and  decision  makers  through  the 
domains  of  HSI  to  determine  where  new  products  present  unacceptable  risks  to  operators, 
maintainers  and  support  personnel.  The  goal  behind  development  of  ABRAHAM  is  to 
provide  time  sensitive,  mission  oriented  subjective  analysis  in  the  absence  of  objective 
data  from  product  manufacturers  and  other  organizations  in  order  to  provide  the  best 
product  to  Soldiers  in  the  field. 

To  achieve  this  goal,  ABRAHAM  uses  queries  to  convert  user  input  into  a 
tailored  questionnaire  covering  all  HSI  domains  for  use  in  alternatives  analysis  and 
product  evaluation.  ABRAHAM  also  provides  basic  analysis  of  responses  and  a  written 
report  describing  areas  of  concern  for  follow-on  analysis  by  REF  analysts  and  contracted 
SMEs. 

D.  VALIDATING  THE  SOLUTION 

Once  the  initial  ABRAHAM  prototype  was  completed,  the  authors  used  the  tool 
to  generate  sample  analyses  on  commercial  and  REF-related  equipment  to  demonstrated 
the  functionality  of  the  tool  and  its  reports.  As  deficiencies  were  identified,  the  authors 
made  changes  to  the  tool’s  design.  (Conceptual  and  programming  challenges  are  fully 
discussed  in  the  Discussion  and  Conclusion  Chapter.) 

ABRAHAM  was  also  demonstrated  by  the  authors  at  three  DoD  related  venues 
(Human  Factors  Engineering  Technical  Advisory  Group  (HFE-TAG),  the  REF,  and  the 
U.S.  Marine  Corps  Operational  Test  and  Evaluation  Command)  to  validate  the  concept 
and  to  obtain  feedback  from  HSI  professionals  and  users  who  might  use  the  tool  in  an 
operational  setting. 
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Upon  completion  of  the  three  demonstration  sessions,  the  authors  presented  the 
findings  TRAC-Monterey.  The  authors  fully  described  the  limitations  of  the  current 
iteration  of  the  tool  and  presented  options  for  future  work  on  the  research.  At  present, 
ABRAHAM  is  one  application  in  a  suite  of  TRAC-Monterey  applications  under 
development  for  use  by  the  REF. 
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III.  THE  MANPOWER  DOMAIN 


A.  MANPOWER  DEFINED 

Historically,  the  manpower  domain  within  the  Department  of  Defense  focused 
solely  upon  the  number  of  personnel  required  to  complete  a  given  mission  (Department 
of  Defense  Directive  1 100.4).  In  May  2005,  the  Office  of  the  Deputy  Chief  of  Staff  G1 
provided  additional  guidance  with  respect  to  whom  the  definition  applies  and  upon  areas 
which  it  focuses.  The  MANPRINT  Handbook  (2005)  expands  the  breadth  of  the 
manpower  domain  including  both  military  and  civilian.  Furthermore,  the  G1 
acknowledged  the  limited  nature  of  defining  manpower  as  only  concerning  the 
completion  of  mission.  In  response  the  G1  proposed  that  manpower  should  be  defined  as 
the  total  manning  throughout  the  system’s  operational  lifecycle  to  include  the 
maintainability,  sustainability,  and  training  necessary  for  the  system  to  perform 
optimally. 

B.  MANPOWER  DOMAIN  IN  HSI 

Within  the  field  of  HSI,  the  manpower  domain  has  encompassed  the 
comprehensive  definition  proposed  by  the  MANPRINT  Handbook  (2005)  (Booher, 
2003).  HSI  practitioners  are  primarily  concerned  with  the  interaction  of  manpower 
within  the  context  of  the  other  six  domains.  However,  this  concern  is  not  only  limited  to 
the  interaction  among  the  other  six  domains  but  also  includes  the  effects  of  manpower 
across  the  entire  lifecycle  of  the  system.  The  HSI  practitioner’s  primary  goal  is  to  ensure 
optimization  by  having  the  right  number  of  people  in  the  right  places  throughout  the 
system.  To  achieve  this  goal,  HSI  practitioners  utilize  tools  such  as  manpower  estimates 
and  workload  analyses  to  provide  alternatives  that  address  a  given  unit’s  capabilities  and 
available  resources.  These  analyses  are  particularly  important  for  the  REF,  which 
typically  imports  new  or  modified  systems  into  existing  circumstances. 

C.  HSI,  MANPOWER,  AND  THE  REF 

At  present,  a  unit  requests  materiel  solutions  from  the  REF  in  order  to  improve 
performance  or  address  an  identified  gap.  As  a  result,  the  REF  analyzes  potential 
alternatives  for  the  unit  to  implement  or  incorporate  into  its  mission.  However,  the 
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materiel  solutions  provided  are  not  necessarily  tailored  to  the  unit’s  available  manning 
resources  and  capabilities.  For  instance,  the  unit’s  current  levels  of  manpower  are  not 
properly  considered  within  the  REF  analysis  or  the  materiel  solutions  provided.  The 
specific  value-added  of  the  HSI  approach  within  the  manpower  domain  is  found  in  the 
systemic  analyses  regarding  the  trade-offs  between  current  levels  of  manpower  and 
required  levels  of  manpower  necessary  when  a  particular  system  is  fielded.  The  HSI 
approach  to  manpower  supplements  existing  methodologies  by  ensuring  that  fielded 
solutions  will  be  optimal  with  current  manpower  levels. 

1.  Goal  of  an  HSI  Manpower  Evaluation 

The  ultimate  goal  of  any  HSI  evaluation  is  to  ensure  that  the  target  audience’s 
capabilities  and  limitations  are  fully  in  the  design  rather  than  as  elements  that  can  adapt 
to  the  system  once  it  is  developed  (Malone  and  Miller,  2003).  A  manpower  analysis  is 
conducted  as  part  of  the  HSI  evaluation  for  a  traditional  acquisition  project  to  determine 
the  most  cost  effective  mix  of  Soldier,  civilian  and  contractor  support  necessary  to 
operate,  maintain,  support,  and  provide  training  for  the  system  in  accordance  with 
Section  2434  of  Title  10,  U.  S.  Code  (Department  of  Defense,  2004).  The  manpower 
determination  can  have  great  influence  on  safety,  design,  training  and  overall  system 
cost. 

Manpower  requirements  for  maintenance  and  support  are  typically  workload 
driven  while  manpower  requirements  for  operators  are  more  frequently  determined  by 
doctrine  (Office  of  the  Deputy  Chief  of  Staff  Gl,  2005).  Manpower  studies  should  be 
conducted  to  ensure:  (1)  design  options  that  reduce  workload  and  increase  program 
affordability  are  given  top  priority;  (2)  total  ownership  costs  are  minimized;  and,  (3) 
future  resources  are  preserved  for  more  efficient  use  in  other  ACAT  programs  (Malone 
and  Miller,  2003).  Among  other  factors,  the  manpower  study  should  include  an  analysis 
of  job  tasks,  operation/maintenance  rates,  associated  workload,  and  operational 
conditions  to  optimize  the  types  and  number  of  people  required  (Department  of  Defense, 
2004). 
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2.  Manpower  Evaluation  in  a  Traditional  Acquisition  Setting 

Most  manpower  analyses  should  begin  with  a  Top  Down  Requirements  Analysis 
(TDRA)  to  identify  each  of  the  relevant  functions  of  the  system  and  which  functions  of 
the  system  should  be  automated,  eliminated,  consolidated  or  simplified  (Malone  and 
Miller,  2003).  For  a  given  operational  condition,  the  TDRA  and  the  functional  and  task 
analyses  should  provide  measures  of  peak  workload,  which  are  usually  stated  in  the  form 
of  average  man-hours  of  work  per  unit  time.  This  workload  data  is  compared  to  a 
workload  standard,  such  as  a  40  hour  work  week,  to  determine  the  number  of  individual 
billets  required  for  the  system  (Malone  and  Miller,  2003). 

This  basic  manpower  computation  must  be  augmented  with  other  factors  that 
affect  Soldiers  as  they  do  their  jobs.  For  example,  environmental  conditions,  fatigue,  or 
cognitive,  physical  or  sensory  overload  in  the  target  audience  may  require  additional 
manpower  to  achieve  a  desired  level  of  performance.  The  analyst  must  also  add  billets 
directed  by  law  or  other  authority,  which  must  be  provided  regardless  of  the  calculated 
workload.  These  “Directed  Billets”  may  be  required  to  provide  a  specific  knowledge, 
skill,  ability  or  qualification  not  normally  found  in  the  target  population  (Malone  and 
Miller,  2003). 

Finally,  the  manpower  analysis  must  look  beyond  the  system  as  an  individual 
item  and  explore  the  effects  the  system  has  from  a  system-of-systems  or  family-of- 
systems  perspective.  While  this  additional  examination  may  identify  commonalities  or 
duplication  between  multiple  systems,  it  will  also  allow  the  analyst  to  examine  the 
distribution  of  workload  among  available  personnel.  It  may  be  determined  that,  while  the 
individual  system  does  not  create  cognitive,  physical  or  sensory  overload,  the  individual 
system  in  concert  with  other  systems  may  create  a  situation  where  more  billets  are 
required. 

The  manpower  analyst  must  be  keenly  aware  that  an  assigned  role  for  human 
performance  may  vary  with  changes  in  operational  conditions.  After  the  minimum  billet 
requirements  are  calculated,  the  manpower  analyst  must  then  examine  other  operational 
conditions  and  scenarios  to  ensure  this  minimum  billet  computation  is  capable  of 
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satisfying  the  manning  requirements,  to  include  how  the  manpower  requirements  will 
impact  training  systems  and  legacy  technologies  found  in  the  unit  (Malone  and  Miller, 
2003). 

Once  the  manpower  functions  and  requirements  are  fully  understood,  the 
manpower  analyst  can  work  with  other  members  of  the  design  team  to  determine  where 
selected  human  tasks  may  be  allocated  to  automation  technologies.  After  allocating  these 
functions  to  either  a  human  or  automation  and  breaking  each  of  these  functions  into  its 
component  tasks,  the  analyst  can  then  determine  the  appropriate  roles  for  the  humans  in 
the  system  (Naval  Sea  Systems  Command,  2003;  Department  of  Defense,  2004). 

Automating  of  tasks  is  enticing  because  it  promises  reduction  in  manpower 
requirements  and  human  error,  and  enhancement  of  situational  awareness.  Used  in 
controlled  circumstances,  automation  can  limit  increases  manpower  requirements  for  new 
systems.  Unfortunately,  automation  is  not  the  magic  wand  that  solves  all  manpower 
issues  because  even  for  an  automated  function  or  task,  the  human  must  still  serve  as 
supervisors,  monitor,  decision  maker,  system  integrator,  or  backup  performer  (Malone 
and  Miller,  2003). 

While  automation  often  reduces  physical  workload,  cognitive  workload  often 
increases.  Operational  experience  and  empirical  research  has  also  shown  that  automation 
does  not  reduce  the  overall  amount  of  errors,  but  instead  has  introduced  new  and  different 
kinds  of  error.  Using  automation  to  enhance  situational  awareness  though  not  necessarily 
a  viable  solution  because  the  technology  often  provides  only  information  and  data.  The 
technology  still  requires  the  human  to  commit  cognitive  resources  to  synthesize  the 
information  and  data  into  knowledge  of  the  tactical  situation  (Kleiner  and  Booher,  2003; 
Malone  and  Miller,  2003). 

Given  the  limitations  of  automation,  technologies  that  can  reduce  the  numbers  of 
humans  required  for  operation,  maintenance  and  support  of  the  system  are  critical.  This 
is  due  to  military  manpower  requirement  being  defined  as  a  zero-sum  environment  due  to 
statutory  constraints  on  military  end  strength.  This  concept  is  critical  because  it  requires 
the  Program  Manager  and  Army  stakeholders  to  make  trade-offs  between  systems  in 
order  to  maintain  force  levels.  If  a  new  system  will  require  more  manpower  than  its 
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predecessor  then  another  system  must  be  identified  as  the  bill  payer  for  the  new  system. 
For  example,  if  a  new  weapons  system  requires  more  Fire  Support  Specialists  Military 
Occupation  Specialty  13F  (MOS  13F)  than  the  Army  inventory  can  support  then  Army 
decision  makers  must  determine  which  MOS  will  lose  manning  to  fill  the  gap  (Kleiner 
and  Booher,  2003;  Malone  and  Miller,  2003). 

In  summary,  the  HSI  domain  of  Manpower  is  concerned  with  determining  the 
optimum  mix  of  military,  civilian  and  contractor  personnel  and  where  possible,  reducing 
the  number  of  people  required  to  safely  and  efficiently  operate,  maintain  and  support  a 
system.  To  achieve  this  mix  and  define  the  manning  reduction  concept  for  the  system, 
the  manpower  analyst  will  normally  conduct  a  TDRA  to  identify  the  roles  of  the  human 
in  the  system,  assign  critical  functions  and  tasks  to  these  roles,  and  examine  the  expected 
performance  of  the  system  through  modeling  and  simulation  of  the  various  operational 
conditions  and  scenarios  in  which  the  system  will  be  employed. 

3.  Specific  Objectives  of  a  Traditional  HSI  Manpower  Evaluation 

•  Define  the  proper  mix  of  military,  civilian,  and  contractor  support  required  to 
efficiently  accomplish  projected  missions  (Malone  and  Miller,  2003). 

•  Document  system  manpower  requirements  and  method  of  requirement  determination 
for  use  in  tradeoff  analysis  (Malone  and  Miller,  2003). 

•  Eliminate  high  frequency,  labor  intensive  tasks  requiring  multiple  people  or  system 
designs  that  increase  manpower  requirements  over  those  of  the  predecessor  systems 
(Air  Force  Human  Systems  Integration  Office,  2003). 

•  Optimize  the  total  manpower  requirements  by  leveraging  technologies  designed  to 
decrease  workload  (Naval  Sea  Systems  Command,  2005). 

•  Identify  mission  tasks,  operational  conditions  and  performance  standards  assigned  to 
humans  in  the  system  (Air  Force  Human  Systems  Integration  Office,  2003). 

•  Identify  environmental  factors  and  operational  conditions  that  may  require  additional 
manpower  to  maintain  acceptable  levels  of  performance  (Department  of  Defense, 
2004). 

•  Identify  constraints  of  predecessor  systems  that  impact  manpower  in  the  new  system 
(Naval  Sea  Systems  Command,  2005). 

•  Determine  the  frequency,  criticality,  and  cognitive,  physical,  and  sensory  demands  of 
mission  functions  and  tasks  to  identify  possible  human/system  overload  situations 
(HSI  VIRTUAL  SYSCOM  WORKING  GROUP,  2005). 
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•  Recommend  process  improvements  and  design  options  to  minimize  workload¬ 
intensive  tasks  and  manpower  levels  required  to  effectively  operate,  maintain  and 
support  the  system  (Malone  and  Miller,  2003). 

•  Identify  policies  or  other  issues  that  impact  attainment  of  manpower  objectives  or  the 
Program  Manager’s  ability  to  successfully  meet  acquisitions  Milestones  (Malone  and 
Miller,  2003). 

•  Develop  policies  to  provide  for  career  progression,  assignment  rotation,  and  combat 
augmentation  of  military,  civilian  and  contractor  personnel  assigned  to  the  system 
(Air  Force  Human  Systems  Integration  Office,  2003). 

•  Validate  the  system's  manpower  quantity  and  quality  requirements  after  completion 
of  system  and  equipment  installations  (Department  of  Defense,  2003). 

4.  A  Practical  Manpower  Evaluation  for  REF  Systems 

Given  the  number  of  systems  that  REF  analysts  examine  and  the  time  constraints 
placed  on  these  examinations,  a  full  TDRA  is  not  practical.  The  REF  analyst  must  be 
able  to  quickly  identify  the  manpower  implications  of  a  new  system  or  technology  and 
make  decisions  concerning  the  feasibility  of  its  implementation.  Because  most  of  the 
REF  products  are  NDI/COTS/GOTS,  the  REF  analyst  does  not  have  the  ability  to 
influence  the  product’s  design  to  increase  automation  or  otherwise  introduce  engineering 
solutions  to  reduce  manpower  requirements. 

The  REF  analyst  must  focus  on  a  sound  assessment  of  whether  the  new  system 
will  require  a  higher,  lower  or  equal  number  of  personnel  as  the  current  technology  or 
doctrinal  solution  demands  and  whether  the  new  system  will  change  the  ranks,  grades  or 
specialties  of  the  target  audience.  To  make  these  determinations,  the  emphasis  for  NDI 
projects  must  be  on  problems  in  the  existing  system/doctrine,  requirements  generated 
from  basic  functional/task  analyses,  requirements  for  reduced  manning,  and  any 
anecdotal,  parametric,  or  subject  matter  expert  data  on  workload/  workload  distribution 
from  previous  or  similar  systems  (Malone  and  Miller,  2003). 

If  the  REF  analyst  determines  that  sufficient  end-strength  is  not  available,  then  the 
analyst  should  be  prepared  to  make  MOS/system  trade  recommendations  to  the  Program 
Manager  if  the  system  is  put  forth  as  a  new  program  of  record  which  could  potentially  be 
fielded  to  the  entire  Army. 
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Suggested  questions  are  provided  to  support  the  manpower  determination  for  the 
REF  analyst.  Specific  metrics  for  additional  REF  manpower  studies  would  include:  time 
on  task,  average  workload,  or  number  of  personnel.  This  discussion  of  the  Manpower 
domain  suggests  several  questions  that  the  REF  analyst  could  pose  to  evaluate  a  proposed 
system  (refer  to  the  Manpower  portion  of  appendix  A  for  questions). 
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IV.  THE  PERSONNEL  DOMAIN 


A.  PERSONNEL  DEFINED 

In  its  most  simplified  form,  the  tenn  personnel  refers  to  “the  people  used  to  fill 
manpower  positions”  (Mobilization  Handbook,  1986).  Manpower  positions  encompass 
both  military  and  civilian  positions.  Policies  governing  DoD  military  personnel  are  set 
forth  in  U.S.  Code  Title  10,  subtitle  1  General  Military  Law,  part  II  Personnel  (U.S.  Code 
Title  10).  In  particular,  Title  10  addresses  recruitment  and  retention  and  outlines  the 
minimum  standards  for  qualified  applicants  as  well  as  the  progression  of  the  members 
through  their  individual  services.  For  the  U.S.  Army,  these  policies  are  further  defined 
by  the  policies  of  the  U.S.  Army  Human  Resources  Command. 

B.  PERSONNEL  DOMAIN  IN  HSI 

For  the  HSI  practitioner,  DoD  personnel  are  the  critical  elements,  which  are  the 
common  link  to  all  systems.  The  Personnel  domain  of  HSI  focuses  on  the  relationship 
between  a  member’s  knowledge,  skills  and  abilities  (KSAs)  and  the  required  KSAs  for 
the  particular  job  in  which  they  are  assigned  to  perform  (Booher,  2003,  p.  387).  The 
Personnel  domain  in  HSI  attempts  to  ensure  that  all  personnel  (i.e.,  operators,  maintainer, 
and  supporter)  are  properly  employed  in  the  appropriate  positions  throughout  their 
service.  Accordingly,  the  HSI  practitioner  ensures  that  all  “personnel  requirements  should 
be  established  consistent  with  the  aptitudes  and  skills  of  the  user  population  that  is 
expected  to  be  in  place  at  the  time  the  system  is  fielded  and  over  the  life  of  the  program” 
(Coast  Guard  Manual,  1994). 

The  HSI  Personnel  domain  also  requires  two  essential  pieces  of  documentation 
(Booher,  2003).  The  first  delineates  the  appropriate  KSAs  required  to  properly  operate 
the  new  system.  This  step  is  important  due  to  its  potential  predictive  ability  ultimately 
eliminate  certain  user  populations.  If  inappropriate  KSAs  are  considered,  practitioners 
may  unknowingly  and  unnecessarily  limit  the  pool  of  qualified  candidates.  The  second 
piece  of  documentation  is  the  accurate  Soldier  personnel  folders,  which  list  previous 
training  and  qualifications  of  potential  users.  Ensuring  that  all  personnel  folders  are  up- 
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to-date  minimizes  potential  waste  of  resources,  such  as  training  unqualified  Soldiers  who 
do  not  possess  the  required  cognitive  capabilities. 

C.  HSI,  PERSONNEL,  AND  THE  REF 

Consistent  with  the  Manpower  domain,  the  REF  is  presented  with  the  challenge 
of  matching  the  system’s  personnel  requirements  to  the  available  resources  within  the 
field  in  which  the  system  will  be  implemented.  In  other  words,  the  personnel  pool  from 
which  qualified  individuals  are  chosen  to  interact  with  the  system  is  already  limited. 
Therefore,  when  addressing  the  necessary  personnel  for  system  implementation,  the  REF 
analyst  does  not  have  the  option  to  alter  the  existing  personnel  structure.  This  approach 
differs  from  the  traditional  acquisition  methods  that  centralized  the  system  (i.e., 
implementing  minimal  modifications  to  the  system)  and  more  often  worked  to 
manipulate  other  variables  to  complement  the  overall  circumstance. 

Due  to  the  REF’s  inability  to  change  personnel  requirements  in  deployed  units, 
the  primary  concern  of  the  REF  analyst  is  to  ensure  the  most  optimal  match  between  the 
system’s  requirements  and  the  KSAs  of  readily  available  personnel.  Essentially,  the  REF 
must  assess  what  is  currently  available  in  the  field  and  estimate  their  ability  to  adapt 
levels  as  it  relates  to  the  system’s  KSA  requirements.  If  a  system  requires  KSAs  that  are 
beyond  or  vastly  different  than  the  competency  levels  of  the  available  personnel,  another 
solution  must  be  sought.  Without  assurance  that  the  product  being  fielded  matches  the 
KSA’s  of  the  available  personnel,  the  system  will  not  reach  optimal  levels  of 
performance.  System  optimization  requires  personnel  (i.e.,  operators,  maintainers,  and 
supporters)  that  understand  the  system. 

1.  Goals  of  an  HSI  Personnel  Evaluation 

Where  a  Manpower  evaluation  is  concerned  with  the  number  and  quality  of  billets 
required  to  accomplish  the  mission,  a  Personnel  evaluation  should  examine  the  people 
who  will  fill  those  billets  to  determine  if  they  possess  the  knowledge,  skills,  abilities, 
aptitudes  and  experience  required  to  successfully  interact  with  the  system  (Department  of 
Defense,  2004).  The  Personnel  evaluation  should  not  only  be  concerned  with  the  current 
military  population  but  should  also  project  into  the  future  to  determine  how  changes  in 
the  population  will  affect  system  performance  (Air  Force  Human  Systems  Integration 
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Office,  2003).  The  Personnel  domain  is  directly  related  to  recruiting,  retention,  training 
system  development,  and  system  design. 

The  Personnel  evaluation  should  build  on  the  Manpower  evaluation  that  includes 
an  analysis  of  job  tasks,  operation/maintenance  rates,  associated  workload,  and 
operational  conditions  to  optimize  the  types  and  number  of  people  required  (Department 
of  Defense,  2004).  The  ultimate  goal  of  the  Personnel  evaluation  is  to  ensure  that  the 
new  system  does  not  require  cognitive,  physical  or  sensory  skills  that  the  military 
population  cannot  support.  Fielding  a  system  that  requires  a  skill  or  ability  that  is  not  in 
the  current  military  population  should  be  considered  a  critical  or  major  risk  to  readiness, 
personnel  tempo,  and  system  funding  requirements  (Office  of  the  Deputy  Chief  of  Staff 
Gl,  2005). 

2.  Personnel  Evaluation  in  a  Traditional  Acquisition  Setting 

As  stated  above,  the  Personnel  evaluation  builds  on  the  Manpower  estimate  and  is 
based  on  the  system  description,  anticipated  skill  requirements  based  on  job  tasks  and 
operational  conditions,  recruiting  and  retention  trends,  and  projected  characteristics  of 
target  occupational  specialties.  In  conjunction  with  the  Manpower  estimate,  a  Target 
Audience  Description  (TAD)  is  developed  to  identify  the  population  that  will  train, 
operate,  maintain,  and  support  the  new  system.  Much  like  the  manpower  estimate,  the 
TAD  is  reconsidered  and  revised  throughout  the  acquisition  process  (Naval  Sea  Systems 
Command  2005;  Office  of  the  Deputy  Chief  of  Staff  Gl,  2005). 

The  TAD  should  include  the  following  infonnation  about  the  military,  civilian, 
and  contractor  force  structure:  military  operational  specialty  (MOS)/series/additional  skill 
indicators  (ASI)  descriptions;  physical  profile  (PULHES)  requirements;  security 
clearance  requirements;  biographical  infonnation;  anthropometric  data;  and  aptitude 
descriptions  as  measured  by  the  Armed  Forces  Vocational  Aptitude  Battery  (ASVAB) 
(Department  of  Defense,  2004).  The  TAD  should  also  address  consideration  for 
combining,  modifying  or  establishing  new  MOS  and  ASI  codes  (Department  of  Defense, 
2004;  Office  of  the  Deputy  Chief  of  Staff  Gl,  2005). 

Personnel  capabilities  are  normally  stated  in  terms  of  knowledge,  skills,  abilities 
(KSAs)  and  are  usually  measured  by  the  percentage  of  the  population  that  meets  the 
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requirement.  There  are  two  well  established  measures  of  personnel  factors  that  can  be 
useful  in  defining  or  classifying  the  population:  ASVAB  and  PULHES  measures. 

The  ASVAB  is  a  Department  of  Defense  multiple  aptitude  test  used  for  selection 
and  classification.  It  has  been  validated  for  training  (Earles  and  Ree,  1992)  and  job 
performance  (Ree  and  Earles,  1994).  The  ASVAB  consists  of  verbal  and  quantitative 
tests,  two  speed  tests  and  technical  knowledge  tests.  The  verbal  and  quantitative  tests  are 
Word  Knowledge  (WK),  Paragraph  Comprehension  (PC),  Arithmetic  Reasoning  (AR), 
and  Mathematics  Knowledge  (MK).  Numerical  Operations  (NO)  and  Coding  Speed  (CS) 
are  the  two  speed  tests.  The  technical  knowledge  tests  are  Electronics  Information  (El), 
Mechanical  Comprehension  (MC),  Auto  and  Shop  Infonnation  (A/S),  and  General 
Science  (GS)  (Powers,  2005). 

The  primary  composite  score  computed  from  the  ASVAB  for  use  by  the  Services 
is  the  Anned  Forces  Qualification  Test  (AFQT).  The  AFQT  score  is  detennined  from 
four  areas  of  the  ASVAB:  Word  Knowledge  (WK),  Paragraph  Comprehension  (PC), 
Arithmetic  Reasoning  (AR),  and  Mathematics  Knowledge  (MK).  The  formula  to  derive 
the  AFQT  “raw  Score”  is  2VE  +  AR  +  MK,  where  VE  (Verbal  Expression)  is  a  scaled 
sum  of  WK  and  PC. 

The  AFQT  “raw  score,”  is  then  converted  into  a  percentile  score.  For  enlistment 
purposes,  AFQT  scores  are  divided  into  the  following  categories:  Category  1  falls 
between  the  93rd- 100th  percentile,  Category  2  falls  between  the  65th-92nd  percentile, 
Category  3A  falls  between  the  50th-64th  percentile,  Category  3B  falls  between  the  31st- 
49th  percentile,  Category  4A  falls  between  the  2 1  st-30th  percentile,  Category  4B  falls 
between  the  16th-20th  percentile,  and  Category  4C  falls  between  the  10th- 15th  percentile. 
Category  5  applicants  fall  below  the  10th  percentile.  Federal  law  mandates  that  nobody 
below  the  10th  percentile  is  eligible  for  selection  and  only  20%  of  applicants  below  the 
30th  percentile  are  eligible  for  service  selection  (Powers,  2005). 

The  PUFHES  is  a  six  factor  profile  system  used  to  set  physical  standards  for  entry 
into  an  MOS.  According  to  Department  of  the  Army  Pamphlet  611-21,  “PUFHES 
identifies  the  broad  physical  demands  of  an  MOS  and  the  physical  ability  required  of  an 
individual  to  perform  the  duties  required  by  the  MOS.  The  physical  profile  serials 
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associated  with  an  individual  MOS  provide  a  more  precise  means  of  matching  individuals 
to  positions”  (Department  of  the  Army,  1999).  The  PULHES  acronym  is  defined  as 
follows: 


P — Physical  capacity  or  stamina. 

U — Upper  extremities. 

L — Lower  extremities. 

H — Hearing  and  ear. 

E — Eyes. 

S — Psychiatric. 

3.  Specific  Objectives  of  an  HSI  Personnel  Evaluation 

•  Define  the  population  of  military,  civilians  and  contractors  including  human 
performance  characteristics  based  on  the  system  description,  operation/maintenance 
rates,  associated  workload,  and  operational  conditions  (Kleiner  and  Booher,  2003). 

•  Identify  Personnel  high  drivers  issues,  to  include: 

•  “personnel  screening  requirements”  (physical  or  mental). 

•  “difficult  jobs  that  increase  aptitude  or  educational  background  requirements” 

•  “qualitative  requirements  that  are  not  in  abundance  in  the  current  and  projected 
recruiting  pool”  (Air  Force  Human  Systems  Integration  Office,  2003). 

•  Determine  if  special  physical  requirements  exist  which  may  limit  the  population 
available  to  operate,  maintain,  and  support  the  system  (Malone  and  Miller,  2003). 

•  Determine  if  special  cognitive  requirements  exist  which  may  limit  the  population 
available  to  operate,  maintain,  and  support  the  system  (Malone  and  Miller,  2003). 

•  Determine  if  special  sensory  requirements  exist  which  may  limit  the  population 
available  to  operate,  maintain,  and  support  the  system  (Malone  and  Miller,  2003). 

•  Review  current  personnel  policy  and  recruitment  trends  to  better  define  the  human 
performance  characteristics  of  the  users  (Department  of  Defense,  2004). 

•  Determine  if  the  new  system  will  lead  to  a  combination,  modification,  or 
establishment  of  new  military  occupational  specialties  or  additional  skill  indicators 
(Malone  and  Miller,  2003). 

•  Verify  that  personnel  availability,  recruiting  issues,  and  anticipated  skill  identifiers 
will  not  degrade  system  operation,  maintenance  or  support  (Department  of  Defense, 
2004). 
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•  Verify  that  the  personnel  requirements  determination  addresses  additional 
requirements  imposed  by  watchstanding  or  duty  cycle  rotation  (Malone  and  Miller, 
2003) 

•  Determine  the  implications  of  reduced  staffing  on  system  performance  (Malone  and 
Miller,  2003). 

•  Explore  technology  implementations  to  reduce  personnel  requirements  throughout  the 
program  lifecycle  (Naval  Sea  Systems  Command,  2005). 

•  Address  specific  factors  that  may  affect  personnel  requirements,  such  as  surge  combat 
requirements  and  expected  duration  of  the  conflict  (Department  of  Defense,  2004). 

4.  A  Practical  Personnel  Evaluation  for  REF  Systems 

Because  the  REF  is  sending  COTS/GOTS/NDI  systems  to  preexisting  units  in  the 
field,  manipulating  knowledge,  skills  and  abilities  is  not  an  option.  The  REF  analyst 
must  focus  on  the  technology  and  compare  the  requirements  for  operation,  maintenance, 
and  support  of  the  equipment  with  the  minimum  standard  allowed  for  an  Army  unit  of  the 
type  that  will  receive  the  equipment.  The  goal  of  this  analysis  should  be  to  ensure  that 
any  equipment  sent  forward  from  the  REF  will  not  overwhelm  the  Soldiers  in  the  field. 

To  make  this  detennination,  the  REF  analyst  should  compare  the  new  technology 
to  any  existing  system/technology  to  determine  the  new  requirements  imposed  by  the 
REF  intervention.  If  there  is  no  existing  system  in  place  to  base  an  anecdotal  or 
parametric  estimate,  a  subject  matter  expert  (SME)  opinion  could  be  used  to  determine  if 
there  are  cognitive,  physical,  or  sensory  skill  requirements  beyond  those  found  in  the 
specified  user  population. 

The  REF  analyst  may  be  able  to  generalize  ASVAB  and  PULHES  scores  to 
classify  the  population  so  that  reasonable  inferences  can  be  made.  For  example,  if  a  REF 
system  is  estimated  to  require  a  CAT  1  AFQT  score  and  PULHES  line  score  of  1 1 1 1 1 1 
then  the  REF  analyst  can  use  this  infonnation  to  determine  that  there  may  be  significant 
difficulty  in  finding  people  to  meet  this  personnel  requirement.  Conversely,  if  a  CAT  3 
AFQT  score  and  PULHES  line  score  of  at  least  1 1 1221  is  required  then  it  may  be  easier 
to  meet  these  personnel  requirements. 
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At  the  end  of  REF  product  evaluation,  the  REF  analyst  should  be  able  to  make 
recommendations  to  the  Program  Manager  for  initial  Personnel  determinations,  including 
MOS/system  trade  recommendations  if  necessary.  The  questions  provided  in  the 
Personnel  section  of  appendix  A  will  assist  in  the  personnel  determination  for  the  REF 
analyst. 
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V.  THE  TRAINING  DOMAIN 


A.  TRAINING  DEFINED 

The  Department  of  Defense  Directive  (DoDD)  1322.18  dated  September  3  2004 
specifically  defines  training  as  the  “instruction  and  applied  exercises  for  acquiring  and 
retaining  the  knowledge,  skills,  and  abilities  (KSA's)  required  to  complete  specific  tasks” 
(Department  of  Defense,  2004).  This  directive  focuses  on  the  methodology  that  leads  to 
the  outcome  -  performing  tasks.  A  few  years  later,  the  Army  expanded  on  the 
Department  of  Defense's  (DoD)  definition  to  further  emphasize  both  the  interaction  of  the 
person  with  some  environmental  variable  and  the  overall  purpose  of  the  tasks.  The 
revised  Army  Field  Manual  (FM)  7-0  (2002)  defines  training  as  the  “process  that  melds 
human  and  material  resources  into  credible,  demonstrable  capability  to  mobilize,  deploy, 
fight,  sustain,  and  win  any  conflict.”  (Department  of  the  Army  FM  7-0,  2002)  The 
ultimate  goal  of  winning  conflicts  provides  a  context  in  which  the  tasks  are  being 
performed.  The  specificity  facilitates  a  common  understanding  about  what  the  training 
should  facilitate  among  all  personnel  from  the  training  developer  to  the  participants  who 
are  the  consumers  of  the  training. 

Within  the  field  of  Human  Systems  Integration  (HSI),  more  precision  was  applied 
to  the  DoD  and  Anny  definitions,  which  led  to  a  more  comprehensive  understanding  of 
the  field  manual's  general  approach.  Booher  (2003),  crafted  a  definition  that  further 
described  the  need  to  not  only  acquire  and  retain  the  necessary  KSAs,  but  also  transfer 
the  acquired  KSAs  to  a  variety  of  environments  such  that  it  transfers  into  the  desired 
performance  (i.e.,  safe  and  effective).  Furthermore,  Booher  expanded  on  training 
methodology  to  include  technology  and  addressed  the  need  to  consider  varying  skill 
levels  as  they  relate  to  the  requirements  of  the  task.  As  such,  the  HSI  field  has 
collectively  defined  training  as  “...promoting  the  safe  and  effective  performance  of 
socio-technical  systems  by  facilitating  the  acquisition,  retention,  and  transfer  of  user 
knowledge,  skills,  and  abilities  through  the  design  of  effective  curricula  and  training 
technologies  and  through  influencing  system  design,  development,  test  and  development 
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in  such  a  way  as  to  effectively  integrate  knowledge  about  requisite  user  skills,  abilities 
and  performance  requirements  throughout  all  phases  of  the  system  lifecycle”  (Archer, 
Headley,  and  Allender,  2003,  p.  440). 

B.  TRAINING  DOMAIN  IN  HSI 

Within  the  field  of  HSI,  training  is  one  of  8  domains.  The  training  domain 
addresses  training  at  two  levels:  strategic  and  tactical.  The  strategic  approach 
underscores  the  importance  of  reflecting  on  the  role  of  training  throughout  the  entire 
existence  of  the  system  -  “from  cradle  to  grave”  -  in  which  tasks  are  performed.  As 
such,  training  developers  need  to  ensure  an  appropriate  training  plan  has  been  designed, 
disseminated  and  continually  evaluated  throughout  the  entire  lifecycle  of  the  system 
(Booher,  2003,  440).  This  comprehensive  approach  addresses  the  needs  and 
requirements  that  may  vary  throughout  the  system  and,  more  importantly,  highlights  the 
importance  of  providing  the  combination  of  KSAs  needed  so  that  personnel  are  equipped 
to  effectively  operate  maintain,  and  support  the  system  over  time  (Booher,  2003,  393). 

On  the  other  hand,  the  tactical  goal  of  training  in  HSI  is  to  develop  effective  and 
efficient  training  instruction  that  promotes  transfer  of  learning  from  the  instructional 
setting  to  the  job.  HSI’s  unique  contribution  to  the  training  area  is  the  consideration  of 
training  transfer.  HSI  recognizes  that  the  overall  effectiveness  of  training  is 
demonstrated  not  only  in  learning  and  retaining  the  content,  but  also  applying  or 
transferring  it  into  effective,  desired  perfonnance.  Previous  studies  REF  have 
demonstrated  that  the  environment  in  which  one  learns  or  trains  has  a  significant  impact 
on  performance,  especially  when  it  is  distinctly  different  than  the  target  work 
environment.  In  general,  the  more  similar  the  training  environment  to  the  work 
environment,  the  more  likely  personnel  will  transfer  their  new  KSAs  to  the  task  at  hand. 
This  is  an  important  consideration  since  the  ultimate  purpose  of  training  is  to  positively 
affect  performance  on  the  job  such  that  it  leads  to  optimally  effective  and  efficient 
outcomes.  Traditionally,  researchers  and  practitioners  in  the  training  domain  have  not 
focused  on  this  aspect  -  likely  due  to  the  ease  with  which  the  line  between  the  training 
plan  and  training  delivery  can  be  blurred  (Booher,  2003,  393).  In  essence,  the  training 
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domain  of  HSI  specifically  addresses  the  training  requirements  for  the  overall  system  and 
refocuses  our  attention  on  the  most  important  outcomes  of  training-perfonnance  on  the 
job. 

C.  HSI,  TRAINING  AND  THE  REF 

Historically,  decision  makers  typically  focused  on  the  gaps  between  current  and 
desired  training  outcomes.  This  analysis  generally  involve  evaluating  the  capacity  of 
current  training  initiatives  and  conducting  a  training  needs  assessment  of  the  new 
systems.  The  gap  is  then  calculated  as  the  difference  between  the  needs  of  the  new 
system  and  the  needs  met  via  current  training  efforts  (Booher,  2003,  393).  Training 
needs  assessment  or  analyses  were  typically  done  by  using  models  such  as  the  ADDIE 
Instructional  Systems  Design  model. 

ADDIE  is  an  instructional  systems  design  (ISD)  model  consisting  of  a  five- 
phased  approach  and  includes  the  following  stages:  the  Analyze,  Design,  Develop, 
Implement,  and  Evaluate.  In  the  Analyze  phase,  the  instructional  problem  is  clarified,  the 
need  and  objectives  are  identified  and  established.  The  Design  phase  is  where  the 
instructional  approach  will  be  developed.  This  phased  is  also  where  the  media  choices 
are  made.  In  the  Develop  phase,  subject  matter  experts  (SMEs)  and  program  developers 
produce  learning  material.  The  Implement  phase  consists  of  putting  into  action  the 
content  developed  within  the  other  phases.  The  final  stage  is  the  Evaluation  phase.  This 
consists  of  an  impartial  evaluation  of  all  material  and  aids.  This  evaluation  identifies 
material  that  should  be  revised. 

Due  to  time  the  limitations  in  which  that  the  REF  operates,  such  models  are  not 
the  best  method  of  analysis.  Constraints  on  time  can  result  in  a  reduction  in  the  breadth 
or  depth  of  the  approach.  The  REF  requires  an  approach  as  compressive  as  the  ADDIE 
but  not  as  time-intensive.  HSI  provides  such  an  approach  within  its  training  domain. 
Additional  HSI  issues  encountered  by  the  REF  are: 

•  Identifying  and  balancing  the  training  tradeoffs  with  other  HSI  domains. 

•  Developing  appropriate  methods  for  training  implementation,  addressing  the 

question  of  embedded  versus  formal  training. 

•  Predicting  training  effectiveness. 

•  Establishing  training  performance  metrics. 
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With  increasing  pressure  to  constrain  budgets,  the  DoD  has  stressed  the 
evaluation  of  training  efforts  or  initiatives  with  respect  to  how  well  the  training  resulted 
in  the  desired  outcomes.  Such  evaluative  efforts  not  only  allow  the  DoD  to  understand 
current  effectiveness  of  training  efforts  but  also  to  track  the  efficacy  of  training 
initiatives.  Therefore,  the  DoD  placed  more  emphasis  on  the  link  between  training  and 
performance.  As  a  result,  preliminary  training  transformation  perfonnance  assessment 
metrics  were  established  (GAO,  2005).  The  development  of  training  performance 
metrics  ensures  that  learners  achieve  the  criteria  or  standards  necessary  for  proper  task 
performance  (GAO,  2005).  Training  effects  are  measured  by  calculating  how  much 
performance  on  a  given  task  improved  as  a  result  of  participating  in  training.  This 
calculation  is  referred  to  as  training  transfer.  Training  efficacy  is  denoted  by  high 
training  transfer  scores. 

Previously  proposed  models  for  training  have  generally  corresponded  to  the 
method  in  which  training  was  implemented.  The  narrow  approach  of  these  models 
manifest  into  limitation,  particularly  for  decision  makers;  thus,  HSI  has  developed  two 
models  to  guide  the  strategic  and  tactical  implementation  of  training.  The  training 
domain,  especially  with  respect  to  the  system  acquisition  model,  encompasses  the 
requirement  design,  testing,  and  deployment  of  the  training  (see  Figure  3).  This  approach 
is  used  at  the  strategic  level  to  implement  training  across  the  lifecycle  of  the  system.  The 
Analysis,  Design,  Development,  Implementation  and  Evaluation  (ADDIE)  model, 
demonstrated  in  Figure  4,  is  used  at  the  tactical  level  of  HSI  for  design  and 
implementation  of  training  methods. 
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HSI  Domains 
Training 


Figure  3.  Schematic  diagram  representing  flow  of  training  expertise  in  HSI 

approach  (Hettinger,  2003). 
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Figure  4.  ADDIE  Model  (Huddle,  C.  2002,  presented  by  Ciavarelli  and  McCauley, 
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1.  Goals  of  an  HSI  Training  Evaluation 

When  the  REF  initiates  a  new  product  or  technology,  the  potential  for  gaps 
between  the  skills  required  to  operate  the  new  product  and  the  skills  possessed  by  unit 
members  is  high  (Niedennan  and  Webster,  1998).  Because  the  REF  is  usually  not 
capable  of  making  substantive  changes  in  the  design  of  the  equipment  or  manipulating 
manpower  and  personnel  requirements,  training  is  the  primary  method  for  closing  this 
gap  (Office  of  the  Deputy  Chief  of  Staff  Gl,  2005).  The  goal  of  the  HSI  Training 
evaluation  is  to  systematically  analyze  formal  and  infonnal  instructional  programs  to 
ensure  that  when  these  programs  are  implemented  skill  levels  increase  and  perfonnance 
in  improved  for  the  total  system  (Air  Force  Human  Systems  Integration  Office,  2003). 

Training  programs  should  be  implemented  to  develop  and  sustain  proficient 
individuals  and  units.  Where  possible,  HSI  seeks  to  leverage  the  increased  levels  of 
performance  to  reduce  lifecycle  costs  (Malone  and  Miller,  2003).  HSI  Training 
evaluations  should  include:  course  design,  development,  and  implementation;  use  of 
simulators  and  training  aids;  and,  embedded  training  capabilities. 

2.  Training  Evaluation  in  a  Traditional  Acquisition  Setting 

For  traditional  acquisitions,  the  Program  Manager  must  design  systems  that  are 
focused  on  the  training  needs  of  the  Combatant  Commander  (Department  of  Defense, 

2004) .  As  with  the  Manpower,  Personnel  and  Human  Factors  Engineering  domains,  an 
effective  HSI  Training  evaluation  begins  with  a  Top  Down  Requirements  Analysis 
(TDRA)  and  full  Target  Audience  Description  (TAD)  (Malone  and  Miller,  2003). 
Analysts  can  identify  required  knowledge,  skills,  and  abilities  by  conducting  task  and 
workload  analyses  for  the  system  while  it  is  under  typical  mission  conditions  and 
compare  these  required  KSAs  to  those  present  in  the  target  population.  Where  shortfalls 
exist  in  the  target  population,  training  analysts  must  determine  if  a  training/instructional 
system  can  be  designed  to  eliminate  the  deficiencies  (Naval  Sea  Systems  Command, 

2005) . 

For  any  program  to  be  effective,  the  Program  Manager  must  design  an  affordable 
but  effective  training  strategy,  execute  the  development  of  the  infrastructure  required  to 
complete  the  training,  manage  the  resources  necessary  to  provide  effective  training 
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throughout  the  system’s  lifecycle,  and  evaluate  the  effectiveness  of  the  training  (Office  of 
the  Deputy  Chief  of  Staff  Gl,  2005).  The  training  strategy  should  focus  on  options  that 
take  advantage  of  technologies  in  the  areas  of  simulation,  embedded  training,  and 
multimedia-based  instruction  to  provide  individual,  team,  and  joint  training  opportunities 
to  operators,  maintainers,  and  support  personnel.  Any  program  requiring  changes  in 
training  infrastructure  require  the  Program  Manager  to  explicitly  identify  schedule, 
technology,  and  funding  risks  that  threaten  program  execution  (Department  of  Defense, 
2003). 


Training  programs  should  consider  the  varying  needs  of  the  following  groups: 
civilians;  contractors;  and  Active,  Guard  and  Reserve  officers;  warrant  officers;  and, 
enlisted  personnel.  Additionally,  training  programs  must  also  consider  pipelines  for 
instructors  and  other  key  personnel.  In  all  cases,  training  requirements  should  be 
established  to  improve  user  capabilities  and  readiness  while  minimizing  lifecycle  costs, 
reducing  the  demand  on  the  training  system,  and  sustaining  optimal  performance  of  the 
system  (Malone  and  Miller,  2003). 

3.  Specific  Objective  of  an  HSI  Training  Evaluation 

•  Identify  knowledge,  skill  and  ability  shortfalls  in  the  target  population 
caused  by  the  introduction  of  new  technology,  doctrine,  and  organizational 
changes  (Kleiner  and  Booher,  2003;  Malone  and  Miller,  2003). 

•  Develop  training  systems  that  allow  operators,  maintainers  and  supporters 
to  interactively  train  and  practice  with  the  system  while  deployed  or 
stationed  remotely  (Malone  and  Miller,  2003). 

•  Evaluate  the  effectiveness  of  training  systems  to  ensure  the  most  efficient, 
cost  effective  development  of  knowledge,  skills  and  abilities  in  the  target 
population  (Department  of  Defense,  2004). 

•  Implement  fault  and  failure  conditions  to  allow  maintainers  and  supporters 
the  ability  to  rehearse  repair  procedures  (Malone  and  Miller,  2003). 

•  Identify  requirements  for  initial  and  refresher  training  needed  to  achieve 
desired  perfonnance  standards  (Malone  and  Miller,  2003). 

•  Promote  training  concepts  that  allow  for  anytime,  anyplace  training  while 
easing  the  burden  on  legacy  training  systems  and  reducing  total  lifecycle 
costs  (Malone  and  Miller,  2003). 

•  Verify  training  requirements  are  based  on  knowledge,  skills  and  abilities 
required  to  perfonn  operational  assignments  (Malone  and  Miller,  2003). 
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•  Verify  training  is  capable  of  supporting  engineering  designs  (Department  of  Defense, 

2004). 

4.  A  Practical  Training  Evaluation  for  REF  Systems 

REF  analysts  must  consider  that  system  failures  can  result  not  only  from  failures 
of  technology  but  also  a  lack  of  training  (Turnage,  1990).  Unfortunately,  not  all  training 
programs  are  effective;  training  programs  may  have  positive,  negative  or  neutral  effects 
(Burke,  1997).  Training  effects,  or  training  transfer,  are  “the  effective  application, 
generalisability  and  maintenance  of  new  knowledge,  skills  and  abilities  to  the  workplace, 
as  a  result  of  undertaking  an  educational  strategy”  (Holton  and  Bates,  2000).  In  order  to 
maximize  the  potential  for  successful  training  interventions,  the  REF  analyst  must 
conduct  an  effective  training  needs  analysis  to  identify  the  knowledge  gaps  created  in  the 
unit  by  the  REF  intervention  (Goldstein,  1993). 

Ideally,  the  REF  would  create  short  duration  training  programs  to  facilitate 
specific  skill  acquisition  by: 

•  Conducting  a  full  analysis  of  a  training  problem  to  establish  a  problem  statement; 

•  Determining  the  goals  of  the  training  intervention;  conducting  a  task/job  analysis 

to  understand  skill  gaps; 

•  Developing  and  classifing  learning  objectives; 

•  Determining  the  instructional  strategy; 

•  Determining  the  best  media  to  deliver  the  training; 

•  Formulating  a  training  system  concept; 

•  Defining  measurement  requirements  for  feedback  and  training  effectiveness 

evaluations  (Eisenstein  2005;  Ciavarelli  and  McCauley  2006). 

The  goal  of  the  training  effectiveness  evaluation  is  to  produce  infonnation 
quantifying  the  transfer  of  training  so  that  decision  makers  can  maintain  or  modify  the 
training  system.  (Bell  and  Waag,  1997) 

The  seminal  work  in  the  realm  of  training  evaluation  was  conducted  by 
Kirkpatrick  (1959).  Kirkpatrick  proposed  that  analysts  should  evaluate  training  systems 
at  four  levels:  the  behavioral  level,  the  learning  level,  the  reaction  level,  and  the  result 
level.  The  behavioral  level  measures  the  trainee’s  satisfaction  with  the  training  and 
training  system.  The  learning  level  measures  the  knowledge,  skills  and  abilities  gained 
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from  training.  The  reaction  level  measures  how  well  the  learning  objectives  of  the 
training  system  are  applied  to  on  the  job  perfonnance.  The  result  level  measures  the 
overall  effect  of  the  training  program  on  the  organization  as  a  whole  (Kirkpatrick  ,1959). 

The  implication  of  the  Kirkpatrick  model  is  that  there  is  a  cause  and  effect 
relationship  in  which  training  leads  to  reactions  and  the  training  reactions  lead  to 
learning.  Learning  leads  to  behavior  changes  which  lead  to  changes  in  the  overall 
organization  (Hamblin,  1974).  Holton  has  criticized  this  implication,  calling  the 
Kirkpatrick  model  little  more  than  taxonomy  as  opposed  to  a  fully  developed  theoretical 
construct  (Holton,  1996).  The  limited  research  conducted  on  the  validity  of  this  linear 
relationship  has  been  inconclusive  (Lefkowitz  1972;  Latham,  Wexley  et  al.,  1975). 

Though  designed  for  Information  Systems  evaluations,  Hamilton  &  Chervany  put 
forth  two  additional  methods  for  effectiveness  evaluation  that  could  be  applied  to  a  rapid 
training  analysis.  The  first,  a  goal-centered  approach,  is  similar  to  the  training  program 
assessment  mentioned  above  in  that  the  analyst  determines  the  objectives  of  the  system  or 
the  units  using  the  system  and  selects  performance  measurements  based  on  this  analysis. 
A  training  evaluation  in  this  case  would  compare  unit  or  individual  performance  to  the 
objectives  of  the  training  system. 

The  second,  and  more  interesting  method,  is  a  system-resource  approach  in  which 
the  analyst  evaluates  system  effectiveness  based  on  the  attainment  of  a  pre-established 
nonnative  state.  Conceptually,  effectiveness  would  be  measured  in  terms  of  resource 
viability  vice  specific  task  or  unit  objectives.  The  system  resource  method  of 
effectiveness  measurement  is  useful  when  the  system  to  be  evaluated  must  be  used  to 
accomplish  tasks  outside  of  the  stated  objectives  or  may  have  consequences  beyond 
mission  accomplishment  (Hamilton  and  Chervany,  1981).  As  an  example  of  the  system- 
resource  approach,  consider  analysis  of  the  use  of  tent  pegs.  The  task  analysis  for  tent 
pegs  may  only  involve  hammering  them  into  the  ground,  but  the  consequence  of  not 
having  the  tent  pegs  could  be  degraded  performance  due  to  increased  troop  fatigue 
because  of  the  lack  of  shelter. 

Unfortunately,  the  REF  analyst  cannot  generally  work  through  these  time 
intensive  processes  due  to  the  constraints  imposed  by  the  customer’s  operational 
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assignment.  No  matter  which  approach  to  conducting  a  training  evaluation  is  used, 
unambiguous  measures  of  training  transfer  are  difficult  to  come  by.  The  training 
evaluation  literature  supports  two  types  of  evaluation:  behavioral  measurement  and  self- 
reported  evaluation.  Behavioral  measurement  is  difficult  to  conduct  because  it  requires 
direct  observation.  While  using  video  cameras  or  facilities  such  as  a  usability  lab  are 
possible,  the  limitations  of  this  method  effectively  constrain  its  use  to  a  laboratory  or 
fixed  simulator  setting  (Haccoun  and  Saks,  1998). 

Self-report  surveys  and  tests  are  the  most  commonly  used  method  to  evaluate  a 
training  program  (Axtell,  Maitlis,  1997).  These  surveys  are  normally  administered  at  the 
completion  of  the  training  course  and  ask  the  trainee’s  subjective  reaction  to  the  training 
in  questions  that  can  either  be  qualified  as  affective  natured  (how  well  did  you  like  the 
instructor)  or  utility  natured  (how  useful  do  you  feel  the  training  program  will  be  to  your 
job).  The  correlation  between  these  reactions  (both  affective  and  utility)  and  on-the-job 
performance  was  found  to  be  better  correlated  than  reaction  measures  and  immediate  and 
retained  measures  of  learning,  but  both  correlations  were  extremely  low  and  not 
recommended  as  a  replacement  for  a  more  complete  analysis  (Alliger  and  Tannenbaum, 
1997). 

Of  course,  the  validity  of  self-reported  data  is  always  a  concern.  One  possible 
way  to  increase  the  confidence  in  self-reported  data  is  to  use  a  multiple  perspective 
evaluation,  similar  to  the  360-degree  feedback  concept  used  for  employee  perfonnance 
evaluations.  By  measuring  the  trainee,  the  supervisor,  and  the  trainee’s  followers,  an 
evaluator  may  be  able  to  get  a  more  precise  evaluation  of  the  effect  of  the  training  on  job 
performance  (Haccoun  and  Saks  1998). 

A  model  that  some  researchers  believe  is  helpful  in  predicting  training  transfer  is 
the  Valance,  Instrumentality,  and  Expectancy  (VIE)  model  (Mathieu  and  Tannenbaum, 
1992;  Haccoun,  1997).  In  theory,  if  the  trainee  believes  that  he  will  be  able  to  use  the 
training  to  improve  his  job  performance  (Efficacy),  believes  the  training  has  value  and  is 
useful  to  his  future  job  perfonnance  (Valance),  and  believes  that  he  will  be  able  to 
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implement  the  lessons  and  techniques  learned  in  training  (Instrumentality)  then  this 
trainee  will  be  more  likely  to  experience  training  transfer  than  a  trainee  that  is  missing 
one  or  more  of  these  elements. 

Utility  reaction  measures  seem  to  measure  the  Value  of  the  course  to  the  trainee. 
Affective  reaction  measures  can  be  designed  to  measure  the  trainee’s  self-efficacy  and 
motivation.  A  study  of  the  trainee’s  work  environment  can  provide  a  measure  of 
Instrumentality. 

Although  they  did  not  comment  specifically  on  the  VIE  model,  Baldwin  and  Ford 
(1988)  put  forth  a  model  that  could  easily  incorporate  the  VIE  model.  The  Baldwin  and 
Ford  model,  Figure  5,  has  three  training  inputs:  Trainee  Characteristics  (which  would 
include  motivation  and  efficacy),  Training  Design  (which  should  consider  factors  that 
make  up  the  Valance  measure),  and  Work  Environment  (which  would  include  factors  that 
contribute  to  Instrumentality)  (Baldwin  and  Ford,  1988). 
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Figure  5.  Model  of  the  Transfer  Process  (Baldwin  and  Ford,  1998) 
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In  the  Baldwin  and  Ford  model,  VIE  factors  have  a  direct  effect  on  learning  and 
retention  while  efficacy  and  instrumentality  factors  directly  impact  learning  and  retention 
and  generalization  and  maintenance  of  the  training. 

Clearly,  individual  differences  such  as  cognitive  ability  and  motivation  impact  the 
transfer  of  training  the  job  performance  (Sein  and  Bostrom,  1989).  There  have  been  a 
limited  number  of  studies  in  which  researchers  have  attempted  to  evaluate  specific 
measures  of  individual  differences  to  predict  training  effectiveness.  There  have  been 
even  fewer  studies  in  which  multiple  factors  of  the  VIE  model  have  been  studied  together 
(Baldwin  and  Ford,  1988).  These  studies  have  met  with  mixed  results  and  do  not  offer 
clear  guidelines  for  use  in  a  predictive  model  at  this  time  (Tan  and  Hall,  2003). 

While  the  scholarly  literature  does  not  provide  specific  guidelines,  there  are  some 
general  concepts  that  should  be  considered.  First,  trainee  self-efficacy  is  important 
(Haccoun  and  Saks,  1998).  Moreover,  the  relevance  of  the  course  sets  the  stage  for 
trainee  motivation  and  desire  to  learn.  Training  relevance  and  trainee  motivation  are 
critical  variables  in  predicting  the  level  of  training  transfer  trainees  self-reported  after  one 
month  (Axtell  and  Maitlis,  1997). 

One  year  removed  from  training,  the  original  motivation  level,  level  of  autonomy 
on  the  job,  and  the  amount  of  training  transferred  after  one  month  are  strong  predictors 
(Axtell  and  Maitlis,  1997).  Support  from  subordinates,  peers,  and  superiors  within  the 
organization  can  also  impact  training  transfer  (Salas  and  Cannon-Bowers,  2001).  As 
Baldwin  and  Ford  (1988)  indicated  in  their  model,  organizational  support  impacts  both 
initial  knowledge  acquisition  and  long-term  transfer  of  training. 

Training  transfer  should  be  evaluated  no  more  than  one  to  three  weeks  following 
the  beginning  of  use  of  the  product  in  a  true  operational  setting  (Mahapatra  and  Lai, 
2005).  Delays  between  training  and  actual  use  can  result  in  significant  knowledge/skill 
decay  (Salas  and  Cannon-Bowers,  2001).  In  one  study,  trainees  were  immediately  able  to 
recall  and  utilize  40%  of  the  training  material  but  only  25%  after  six  months  and  15% 
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after  being  removed  from  training  for  one  year  (Burke,  1997).  If  there  is  an  expectation 
of  a  long  delay  between  training  and  use,  refresher  training  is  recommended  (Antle  and 
Barbee,  2003). 

No  matter  how  comprehensive  the  training  program,  it  is  not  reasonable  to  believe 
that  a  training  intervention  alone  will  guarantee  changes  in  work  practice  or  long  term 
organizational  outcomes  (Baldwin  and  Ford,  1988;  Sein  and  Bostrom,  1989).  To  help 
maximize  training  transfer  and  the  effective  use  of  REF  equipment,  it  may  be  necessary 
to  inform  trainees  of  specific  situations  that  may  limit  the  use  of  new  skills  and  teach 
trainees  how  to  recognize  possible  barriers  to  effective  use  of  the  training  concepts.  The 
REF  must  also  supply  the  trainee  with  methods  to  overcome  unfavorable  environmental 
conditions,  and  encourage  the  trainee  to  use  these  mediation  methods  in  difficult 
situations  (Marx,  1982). 

Finally,  training  is  believed  to  be  most  effective  when  the  training  program 
“instills  general  principles,  and  a  multiplicity  of  practical  examples  and  demonstrations  of 
the  learning  points”  through  maximum  repetition  of  experiential  learning  vice  passive 
learning  techniques  (Haccoun  and  Saks,  1998). 

Without  the  ability  to  fully  design  and  implement  its  own  instructional  system  or 
manipulate  manpower  and  personnel  requirements,  the  REF  must  capitalize  on  the  effects 
of  training  by  preparing  Soldiers  to  obtain  the  highest  possible  of  transfer.  Without 
considering  individual,  organizational,  and  training  design  factors,  it  will  be  difficult  for 
the  REF  to  make  informed  decisions  as  to  whether  the  limitations  of  a  piece  of  equipment 
are  due  to  technological  failures  or  due  to  poor  training  design  or  transfer.  This 
discussion  of  the  Training  domain  suggests  several  questions  that  the  REF  analyst  could 
pose  to  evaluate  a  proposed  training  system  (refer  to  the  Training  portion  of  appendix  A 
for  questions). 
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VI.  THE  HABITABILITY  DOMAIN 


A.  HABITABILITY  DEFINED 

The  Habitability  domain  of  HSI  refers  to  “those  issues  associated  with  the  living, 
sleeping  and  eating  within  the  confines  of  the  system”  (Miller,  N.  L.,  Crowson,  J.,  and 
Narkevicius,  J.  M.,  2003,  p.726). 

B.  HSI,  HABITABILITY,  AND  THE  REF 

Since  most  of  the  systems  procured  through  the  REF  will  not  involve  evaluations 
of  living/working  space,  temperature,  sanitation  or  ventilation,  the  habitability  evaluation 
for  the  REF  will  likely  be  limited  to  two  important  factors:  (1)  improving  Soldier  quality 
of  life  and  (2)  providing  initial  consideration  for  habitability  from  a  system  of  systems 
perspective. 

In  the  area  of  quality  of  life,  the  REF  analyst  should  consider  how  the  human 
machine  interface,  training  provisions,  and  doctrinal  implementation  of  the  system  affect 
the  Soldier’s  performance  of  his/her  duties,  and  enhance  job  satisfaction,  job 
enlargement,  and  enrichment.  Where  applicable,  habitability  trade-offs  should  reduce 
physical  and  psychological  stress  to  ensure  that  Soldiers  are  rested,  vigilant,  motivated 
and  unimpaired  (Malone  and  Miller,  2003). 

The  REF  analyst  should  also  consider  the  new  system  as  a  part  of  a  system  of 
systems.  This  evaluation  should  determine  the  impact  that  the  new  system  will  have  on 
other  systems  with  which  it  may  be  employed.  More  specifically,  when  the  REF  equips  a 
unit  with  a  new  piece  of  gear,  the  analyst  should  consider  how  the  new  equipment  may 
affect  the  habitability  of  systems  used  to  transport  the  Soldiers  in  the  field  (Department  of 
Defense,  2003). 

1.  Goals  of  an  HSI  Habitability  Evaluation 

The  HSI  analyst  conducts  a  habitability  evaluation  to  identify  and  address  any 
factors  related  to  living  or  working  conditions  that  might  be  required  for  extended  system 
employment.  The  specific  goal  of  a  habitability  evaluation  is  to  “optimize  mission 
readiness,  crew  morale,  professional  development,  retention,  and  recruitment  that  support 

system  performance”  (Naval  Sea  Systems  Command,  2005). 
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Habitability  factors,  including  Soldier  morale  and  comfort,  quality  of  life  issues, 
and  physical  environment,  cannot  be  endlessly  sacrificed  for  other  engineering  priorities 
or  HSI  domain  requirements  without  eventually  impacting  personnel  performance,  unit 
readiness,  and  recruiting  and  retention  (Department  of  Defense,  2003;  Malone  and  Miller, 
2003;  Department  of  Defense,  2004).  A  well-executed  implementation  of  habitability 
factors  can  significantly  ease  requirements  in  the  HSI  domains  of  Manpower  and 
Personnel. 

2.  Habitability  Evaluation  in  a  Traditional  Acquisition  Setting 

A  comprehensive  habitability  examination  may  be  more  applicable  for  large 
systems  such  as  the  Navy’s  ship  procurement  and  large  ground  systems  such  as  the 
M1A1  tank.  In  such  systems,  the  HSI  professional  would  work  with  engineers  to 
establish  acceptable  system  standards  for  living/working  space,  temperature,  noise, 
lighting,  ventilation  and  sanitation  (where  applicable).  In  establishing  these  standards, 
there  may  be  significant  overlap  with  the  HSI  domains  of  System  Safety,  Environment, 
Safety  and  Occupational  Health  (ESOH),  and  Soldier  Survivability  (Department  of 
Defense,  2004). 

As  Doctrine,  Organization,  and  Leadership  become  more  solidified  the  HSI 
professional  should  also  consider  the  personal  services  required  by  the  Soldiers  operating, 
maintaining,  and  supporting  the  system.  These  personal  services  may  include  provisions 
for  messing,  religious  services,  and  mail  or  other  methods  of  communication.  These 
habitability  detenninations  would  be  geared  at  improving  quality  of  life  issues  during 
sustained  operations  that  would  have  a  direct  impact  on  retention  and  recruiting 
requirements.  This  discussion  of  the  Habitability  domain  suggests  several  questions  that 
the  REF  analyst  could  pose  to  evaluate  the  Habitability  domain  (refer  to  the  Habitability 
portion  of  appendix  A  for  questions). 

3.  Specific  Objectives  of  an  HSI  Habitability  Evaluation 

•  Identify  habitability  requirements  (physical  environment,  requirements  for 
personnel  services,  and  living  conditions)  required  to  sustain  system  performance, 
promote  personnel  recruiting/  problems,  and  maintain  quality  of  life  (Malone  and 
Miller,  2003). 


50 


•  Analyze  the  impact  of  ventilation,  noise,  lighting,  messing,  berthing,  sanitation, 
and  quality  of  life  initiatives  on  sustaining  performance  requirements  and 
sustaining  mission  effectiveness  (Malone  and  Miller,  2003). 

•  Ensure  that  stowage  of  gear  does  not  impact  crew  habitability  (Malone  and 
Miller,  2003). 

•  Verify  safety  issues  are  identified  and  corrected,  shielded  or  guarded  (Malone  and 
Miller,  2003). 

•  Verify  that  system  layout  allows  for  comfortable  operation,  maintenance  and 
support  of  the  system  (Malone  and  Miller,  2003). 
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VII.  SURVIVABILITY 


A.  SURVIVABILITY  DEFINED 

The  Department  of  Defense  (DoD)  has  addressed  survivability  from  the 
perspective  that  personnel  endurance  is  essential  to  mission  accomplishment.  The  DoD 
acknowledges  that  in  combat,  risks  include  accidental  death  of  Soldiers,  nuclear, 
chemical,  and  biological  hazards,  environmental  limitations,  and  the  necessity  for 
immediate  evacuation  during  life-threatening  system  conditions.  Given  these  particular 
threats,  “the  PM  must  address  the  system  and  potential  supplements  in  order  to  optimize 
the  survivability  of  the  personnel  as  well  as  the  system  itself’  (Department  Of  Defense 
5000.2,  2003). 

Army  Regulation  (AR)  70-75  (2005)  supplements  the  DoD’s  guidelines  with  a 
specific  distinction  between  two  elements  of  survivability:  Soldier  and  system.  Soldier 
survivability  focuses  on  the  ability  of  the  Soldier  to  not  only  survive  when  carrying  out  a 
mission  but  also  persist  with  effective  performance  within  decreased  system  capabilities. 
Mission  accomplishment  relies  upon  the  survivability  of  the  system  as  well. 
Survivability  is  enhanced  when  systems  are  designed  to  endure  (i.e.,  withstand)  hostile 
environments  or  avoid  them  all  together. 

B.  SURVIVABILITY  DOMAIN  IN  HSI 

HSI  addresses  survivability  in  a  manner  that  considers  both  personnel  and  system 
factors  simultaneously.  The  DoD  5000.2  (2003)  states  “the  PM  shall  address  personnel 
survivability  issues  including  protection  against  fratricide,  detection,  and  instantaneous, 
cumulative,  and  residual  nuclear,  biological,  and  chemical  effects;  the  integrity  of  the 
crew  compartment;  and  provisions  for  rapid  egress  when  the  system  is  severely  damaged 
or  destroyed.  The  PM  shall  address  special  equipment  or  gear  needed  to  sustain  crew 
operations  in  the  operational  environment”  (Department  of  Defense  5000.2c,  2003,  p. 
33).  Although  similar,  Zigler  and  Weiss’  HSI  approach  to  survivability  slightly  differs 
from  the  DoD  and  Army  approach.  HSI  examines  the  Soldier  and  system  as 
interconnected  variables.  Because  HSI  centralizes  the  impact  of  the  human  within  the 
system,  HSI  analysts  view  survivability  as  “...man-machine  interaction  and  the  effects  as 
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a  whole  with  the  ultimate  focus  on  the  survival  of  the  human”  (Zigler  and  Weiss,  2003 
p.627).  Therefore,  personnel  survivability  is  intricately  interwoven  with  the  system’s 
survivability.  According  to  Malone  and  Miller  (2003),  survivability  of  personnel 
depends  upon  the  design  of  a  system  that  reduces  “...the  risk  of  fratricide,  detectability, 
and  probability  of  being  attacked”  (p.  38).  Ultimately,  such  actions  allow  personnel  to 
accomplish  the  mission,  endure  hostile  environments,  and  mitigate  the  effects  of  chronic 
illness,  disability,  death,  or  abort  the  mission  (Malone  and  Miller,  2003).  The  HSI 
perspective  aligns  with  the  six  principals  of  the  military  personnel  domain  (Zigler  and 
Weiss,  2003): 

•  Reduce  fratricide 

•  Reduce  detection 

•  Reduce  probability  of  being  attacked 

•  Reduce  damage 

•  Minimize  injury 

•  Reduce  mental  and  physical  fatigue 

C.  HSI,  SURIVIABILITY,  AND  THE  REF 

The  mission  of  the  REF  is  to  “...provide  operational  commanders  with  rapidly 
employable  solutions  to  enhance  lethality,  survivability  and  force  protection  through 
insertion  of  COTS-GOTS  (Equip)  and  Future  Force  technologies  (Insert)  while  infonning 
Army  stakeholders  (Assess)  to  remain  ahead  of  an  adaptive  enemy”  (www.ref.army.mil). 
HSI  analysts  contribute  to  this  mission  by  conducting  analyses  that  optimize  the 
survivability  of  the  personnel  and  the  system  by  suggesting  alternatives  that  mitigate  the 
vulnerabilities  and  threats  that  front-line  troops  often  confront.  By  applying  an  analytical 
approach  that  views  personnel  and  system  survivability  as  interwoven  factors,  HSI 
analysts  ensure  optimal  performance.  Stated  differently,  system  survivability  is  not 
optimal  if  personnel  cannot  endure  environmental  or  hostile  conditions.  Therefore,  the 
HSI  practitioner  must  be  concerned  about  both  areas  of  survivability;  this  provides  the 
most  comprehensive  and  valuable  approach. 

1.  Goals  of  an  HSI  Survivability  Evaluation 

Under  peacetime  operating  conditions,  reduction  of  injury  and  death  is  the  focal 

point  in  the  domains  of  Human  Factors  Engineering,  Habitability,  and  Environment, 
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Safety  and  Occupational  Health.  The  Survivability  domain  is  focused  on  combat  and 
attempts  to  reduce  an  obvious  scenario  (Kleiner  and  Booher,  2003).  As  an  example  of 
this  distinction,  consider  an  obvious  scenario.  Under  peacetime  conditions,  analysts  in 
Human  Factors  Engineering,  Habitability,  and  Environment,  Safety  and  Occupational 
Health  could  design  a  tent  that  best  protects  its  occupants  from  the  weather:  they  may 
give  no  consideration  to  the  color  of  the  tent.  This  tent  that  meets  all  peacetime 
requirements  might  be  problematic  under  combat  conditions  if  it  were  colored  Dayglo 
Yellow;  survivability  analysts  would  make  recommendations  to  correct  this  deficiency  by 
selecting  a  color  that  would  be  more  difficult  for  the  enemy  to  detect. 

The  goals  of  an  evaluation  in  the  survivability  domain  are  to  influence  system 
design  to  reduce  the  likelihood  that  an  enemy  might  detect  and  attack  our  Soldiers  and  to 
increase  the  ability  of  Soldiers  to  withstand  injury  when  attacked.  During  the 
survivability  evaluation  consideration  must  also  be  given  to  the  risk  of  fratricide  and 
measures  to  prevent  fratricide  episodes,  both  those  inflicted  by  other  units  on  the  system 
being  evaluated  and  those  inflicted  by  the  system  being  evaluated  on  other  units 
(Department  of  Defense,  2004). 

While  the  concept  of  survivability  should  be  applied  to  both  human  and  system, 
the  HSI  domain  of  survivability  should  be  focused  primarily  on  survivability  of  the 
human  (Kleiner  and  Booher,  2003).  Specifically,  the  survivability  analyst  should 
examine  the  instantaneous,  cumulative,  and  residual  effects  of  chemical,  biological, 
radiological,  nuclear,  and  explosive  (CBRNE)  attacks  on  the  ability  to  continue 
operations  without  exposing  Soldiers  to  acute  chronic  illness,  disability,  or  death 
(Department  of  Defense,  2004).  The  survivability  evaluation  should  also  consider 
antiterrorism/force  protection  (AT/FP)  measures,  the  impact  of  Mission  Oriented 
Protective  Posture  (MOPP)  gear  or  other  protective  equipment  on  operations,  reduction 
of  mental  and  physical  fatigue,  crew  compartment  integrity  and  rapid  egress  capabilities 
for  systems  that  have  been  damaged  or  destroyed  (Air  Force  Human  Systems  Integration 
Office,  2003;  Malone  and  Miller  2003;  Department  of  Defense,  2004;  Office  of  the 
Deputy  Chief  of  Staff  Gl,  2005). 
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2.  Survivability  Evaluation  in  a  Traditional  Acquisition  Setting 

At  the  beginning  of  a  traditional  acquisition  program  (pre-Milestone  B),  the  new 
system’s  design,  operating  environment,  and  doctrine  may  lack  sufficient  detail  to  permit 
critical  analysis.  Throughout  the  acquisition  lifecycle  the  Program  Manager  must 
implement  an  iterative  Personnel  Survivability  program  that  reduces  detection  and  attack 
by  the  enemy,  minimizes  serious  injury  by  operator  and  maintenance  personnel,  and 
prevents  fratricidal  incidents  (Department  of  Defense,  2004).  As  concept  refinement  and 
system  development  become  more  mature,  survivability  analysis  can  become  more 
robust. 

Survivability  assessment  is  primarily  conducted  in  the  Low  Rate  Initial 
Production  phase.  During  this  phase,  the  system  will  undergo  initial  operational  test  and 
evaluation  and  live  fire  testing.  Of  course,  not  all  aspects  of  survivability  can  be 
addressed  in  test  and  evaluation,  and  the  Program  Manager  should  supplement  test  and 
evaluation  with  modeling  and  simulation. 

To  fully  address  survivability,  the  domain  is  generally  divided  into  seven 
components:  (1)  Reducing  Fratricide,  (2)  Reducing  Detectability,  (3)  Reducing 
Probability  of  Attack,  (4)  Minimizing  Damage,  (5)  Minimizing  Injury,  (6)  Minimizing 
Physical  and  Mental  Fatigue,  and  (7)  Surviving  Extreme  Environments  (Malone  and 
Miller  2003;  Department  of  Defense,  2004).  These  seven  components  can  be  addressed 
through  methods  as  basic  as  use  of  camouflage,  smoke,  and  deception  techniques  that 
make  the  system  appear  like  a  target  of  lesser  value  or  through  more  technological 
interventions  such  as  signal  distortion,  Identification  of  Friend  or  Foe  (IFF)  technology, 
and  countermeasure  systems.  Additionally,  Program  Managers  may  also  be  able  to 
influence  survivability  through  training,  doctrinal  change,  human  factors  engineering 
(Malone  and  Miller  2003;  Department  of  Defense,  2004).  Chapter  Six  of  the  Defense 
Acquisition  Guidebook  provides  a  more  complete  description  of  these  seven  survivability 
components  (Department  of  Defense  Acquisition  Guidebook,  2004). 
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3.  Specific  Objectives  of  an  HSI  Survivability  Evaluation 

•  Identify  doctrine,  training  and  technological  initiatives  to  improve  the  ability  to 
correctly  identify/be  correctly  identified  by  friendly  forces  (Malone  and  Miller, 
2003). 

•  Identify  specific  methods  to  reduce  the  probability  of  detection  and  attack  by  enemy 
forces  (Malone  and  Miller,  2003). 

•  Identify  personal  protective  gear/equipment  required  to  minimize  injury,  illness  and 
death  caused  by  instantaneous,  cumulative  and  residual  CBRNE  weapon  effects 
(Malone  and  Miller,  2003). 

•  Maximize  crew  compartment  integrity  during  combat  operations  (Malone  and  Miller, 
2003). 

•  Ensure  viability  of  planned  rapid  egress  methods  given  severe  system 
damage/destruction  (Malone  and  Miller,  2003)  . 

•  Verify  the  design  of  personnel  survivability  alarms  and  alerts  effectively  convey 
required  information  to  crew  members  (Malone  and  Miller,  2003). 

•  Ensure  damage  control  tasks  have  been  assessed  after  completion  of  final  equipment 
and  system  installations  (Malone  and  Miller,  2003). 

•  Continually  evaluate  the  following  Survivability  high  drivers  for  changes  in  risk 
assessment  and  system  survivability  (Air  Force  Human  Systems  Integration  Office, 
2003): 

•  New  weapons  technology. 

•  Weapons  designed  for  crew  incapacitation. 

•  Weapons  designed  to  inflict  damage  on  the  system. 

•  New  Radar  technologies  that  increase  detectability. 

•  Adverse  changes  in  operating  environments. 

•  Changes  in  own/enemy  doctrine. 

4.  A  Practical  Survivability  Evaluation  for  REF  Systems 

The  HSI  domain  of  survivability  aligns  with  two  of  the  REF’s  primary  focus 
areas:  survivability  and  force  protection.  It  is  likely  that  many  of  the  REF  NDI  solutions 
will  be  developed  exclusively  for  use  in  the  civilian  sector,  and  as  such,  will  not  have 
been  evaluated  with  the  demands  of  the  combat  environment  in  mind.  Fortunately,  the 
REF  analyst  will  be  supported  by  the  Army  Test  and  Evaluation  Command  (ATEC), 
which  will  provide  a  basic  safety  analysis  for  the  equipment. 
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While  ATEC  will  provide  the  REF  with  a  limited  objective  evaluation  of  the 
system,  the  REF  analyst  can  supplement  this  evaluation  with  a  subjective  global 
assessment  to  identify  critical  survivability  issues.  To  support  any  commercial 
survivability  analysis  conducted,  the  REF  analyst  should  consider  the 
“availability/adequacy  of  protection  systems  and  devices,  expected  human  perfonnance 
wearing  protective  ensembles,  adequacy  of  countermeasures,  and  adequacy  of  the  design 
for  survivability”  (Malone  and  Miller,  2003  p.  124).  In  order  to  conduct  this  overarching 
subjective  evaluation,  the  REF  analyst  must  determine  early  in  the  equipping  process  the 
doctrine  and  environment  against  which  systems  must  be  evaluated. 

The  overarching  subjective  evaluation  might  use  a  system  similar  to  the  risk 
management  matrix  in  which  the  severity  of  the  consequence  is  compared  to  the 
probability  of  occurrence  (Clark  and  Goulder,  2002).  To  standardize  measures  of 
severity,  the  REF  analyst  should  use  the  following  delineation:  “[Category  1]  - 
Catastrophic  -  may  cause  death  or  system  loss;  [Category  2]  -  Critical  -  may  cause 
severe  bodily  injury,  severe,  occupational  illness,  or  major  system  damage;  [Category  3] 
-  Marginal  -  may  cause  minor  bodily  injury,  minor  occupational  illness,  or  minor  system 
damage;  and  [Category  4]  -  Negligible  -  may  cause  less  than  minor  bodily  injury, 
occupational  illness,  or  minor  system  damage”  (Clark  and  Goulder,  2002  p.  90). 

No  organization  is  capable  of  addressing  every  survivability  concern,  and  it  is 
appropriate  for  the  REF  analyst  to  consider  contingency  management  techniques  and 
expected  performance  given  damage  to  selected  systems.  If  a  deficiency  is  noted  through 
any  method  of  evaluation  then  the  REF  analyst  should  make  recommendations  to  address 
the  deficiency  through  system  design,  training,  or  other  protective  measures  (Malone  and 
Miller,  2003).  Any  decision  to  accept  potential  risks  should  be  formally  documented  by 
the  REF  analyst  and  forwarded  to  the  decision  maker  for  consideration  (Naval  Sea 
Systems  Command,  2005). 

This  discussion  of  the  Survivability  domain  suggests  several  questions  that  the 
REF  analyst  could  pose  to  evaluate  the  Survivability  domain  (refer  to  the  Survivability 
portion  of  appendix  A  for  questions). 
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VIII.  THE  ENVIRONMENT,  SAFETY,  AND  OCCUPATIONAL 

HEALTH  (ESOH)  DOMAIN 


A.  ESOH  DEFINED 

The  Department  of  Defense  Instruction  5000.2  (2003)  addresses  the  ESOH 
domain  in  three  distinct  sections.  The  environmental  aspects  of  this  instruction  direct  the 
Program  Manager  to  analyze  the  impact  of  the  system  on  its  surrounding  environment  as 
it  relates  to  safety.  The  safety  portion  of  this  instruction  instructs  the  PM  to  evaluate  all 
safety  programs  that  apply  to  the  system,  as  well  as  the  impact  on  the  operators  and 
maintainers.  Finally,  the  occupational  health  section  of  this  instruction  investigates 
whether  the  well  being  of  the  operators  and  maintainers  is  continually  considered. 

Due  to  the  general  nature  of  the  DoD  instruction  for  safety,  the  Army  developed 
detailed  policies  and  published  them  Army  Regulation  (AR)  385-10.  The  Department  of 
Labor  (DoL)  standards  were  used  as  guiding  principles  for  minimum  standards.  These 
policies  and  procedures,  are  designed  to  prevent  injury  or  damage  to  Army  personnel  or 
property  (AR  385-10,  2002). 

AR  385-16,  System  Safety  Engineering  and  Management,  provides  the  system 
safety  engineering  and  management  guidelines  for  Army  safety  throughout  the 
acquisition  process  (Army  Regulation  385-16,  2004).  Specifically,  this  regulation 
prescribes  policies  and  procedures  that  ensure  systems  and  facilities  hazards  and  their 
associated  risks  are  identified  throughout  the  lifecycle  of  the  system.  AR  385-16  also 
ensures  that  safety  programs  are  implemented  for  all  nondevelopmental  items  (NDI). 
Therefore,  the  same  guiding  principles  and  minimum  standards  apply  for  NDIs  such  that 
“...application  of  system  safety  should  start  immediately  after  definition  and 
identification  of  need”  (AR  385-16,  2004). 

B.  ESOH  DOMAIN  IN  HSI 

Unlike  other  domains  within  HSI,  the  ESOH  domain  is  more  mature  due  to  it 
being  based  on  the  DoL  ESOH  guidelines.  The  HSI  practitioner’s  main  focus  within  the 
ESOH  domain  is  to  ensure  that  all  safety  regulations  are  being  adhered  to  throughout  the 
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system’s  lifecycle.  In  addition,  the  HSI  practitioner  ensures  that  the  tradeoff  between  the 
various  domains  does  not  create  circumstances  in  which  the  safety  factors  fall  below 
minimum  standards. 

C.  HSI,  ESOH,  AND  THE  REF 

Due  to  the  nature  of  the  systems  the  REF  employs,  the  REF  analyst  must  ensure 
that  safety  criteria  are  strictly  adhered  to  from  the  outset  of  the  REF  acquisition  process. 
Not  only  should  the  minimum  safety  standards  be  applied,  but  also  stricter  standards 
should  be  considered  due  to  the  environment  in  which  the  system  will  be  employed. 

1.  Goals  of  an  HSI  Environment,  Safety  and  Occupational  Health 
(ESOH)  Evaluation 

The  goal  of  an  Environment,  Safety  and  Occupational  Health  (ESOH)  evaluation 
is  to  protect  the  operators,  maintainers,  supporters  and  their  environments  from 
unnecessary  risks  that  may  result  in  injury,  illness,  or  death  (Office  of  the  Deputy  Chief 
of  Staff  Gl,  2005).  ESOH  evaluations  differ  from  Survivability  evaluations  in  that  the 
Survivability  evaluation  is  based  on  operations  in  a  combat  or  threat  environment 
(Malone  and  Miller,  2003).  As  an  example,  an  ESOH  evaluation  of  a  helicopter  may 
focus  on  environment,  safety,  and  health  issues  of  standard  operations  whereas  a 
survivability  evaluation  would  examine  how  these  issues  are  complicated  by  enemy 
detection,  attack,  and  lethality  capabilities. 

ESOH  evaluations  are  generally  divided  into  two  sections:  System  Safety  and 
Health  Hazards.  System  Safety  examinations  seek  to  minimize  or  eliminate  factors  that 
may  lead  to  human  or  machine  errors/failures  that  will  potentially  result  in  a  mishap 
causing  injury  or  death.  A  Health  Hazard  examination  explores  system  design  and 
concept  of  operations  to  identify  factors  that  will  result  in  significant  risk  of  injury, 
illness,  or  death  (Malone  and  Miller,  2003). 

ESOH  evaluations  can  result  in  changes  to  system  design  or  operating  concept 
that  minimize  manpower,  personnel,  and  training  requirements  (Air  Force  Human 
Systems  Integration  Office,  2003).  The  Navy’s  Mk  V  Special  Operations  Craft 
demonstrates  this  concept.  The  craft  experiences  slamming  rated  at  up  to  20g  which 
results  in  perfonnance-degrading  injury  to  30  percent  of  Special  Warfare  Combat 
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Crewmen  (McCarter,  2004).  If  the  effects  of  the  ship’s  impacts  could  be  minimized  then 
manpower  and  personnel  initiatives  to  replace  and  train  this  30  percent  of  crewmen  could 
be  eliminated  over  the  lifecycle  of  the  system. 

An  effective  ESOH  evaluation  should  result  in  minimizing  both  the  potential  for 
mishap  and  the  consequences  of  mishaps  that  are  unavoidable.  Minimizing  risk/impact 
should  increase  operational  readiness  and  mission  effectiveness  and  reduce  damage  to  the 
environment  and  loss  of  equipment  (Air  Force  Human  Systems  Integration  Office,  2003). 

2.  Environment,  Safety  and  Occupational  Health  (ESOH)  Evaluation  in 
a  Traditional  Acquisition  Setting 

Throughout  the  acquisition  lifecycle  the  program  manager  must  implement  an 
iterative  ESOH  program  that  will  minimize  injury,  illness,  and  death  of  operators, 
maintainers,  and  support  personnel  and  minimizes  damage  to  equipment  and  environment 
(Department  of  Defense,  2004).  Specifically,  the  program  manager  should  address  risks 
by  concentrating  on  ESOH  concerns  early  in  the  systems  engineering  process,  identifying 
a  method  for  tracking  ESOH  progress,  and  developing  a  schedule  for  the  system  to  be 
come  fully  compliant  (Department  of  Defense,  2003).  The  ESOH  program  should 
consider  problematic  system  design  features;  use  of  hazardous  materials;  exposure  to 
toxic  chemicals;  environmental  stressors  such  as  thermal,  vibration,  noise,  and  electrical 
and  other  fluid  discharge  (Malone  and  Miller,  2003). 

The  ESOH  evaluation  should  examine  all  operations,  support,  and  maintenance 
situations  that  Soldiers  should  expect  to  experience  during  the  lifecycle  of  the  system. 
Early  in  the  acquisition  process,  it  is  often  difficult  to  establish  measurable  safety  and 
health  requirements  due  to  the  preliminary  nature  of  the  system  design  and  operational 
concepts.  Requirements  at  this  stage  of  the  process  are  typically  generated  from  lessons 
learned  from  human  factors  issues  in  comparable  or  predecessor  systems.  Later  in  the 
acquisition  process,  requirements  can  be  altered  to  meet  system-specific  issues  that  can 
be  analyzed  using  MIL-STD-882  series  and  other  safety  guidelines  (Air  Force  Human 
Systems  Integration  Office,  2003;  Malone  and  Miller,  2003). 

When  Manpower,  Personnel,  Training,  or  Human  Factors  Engineering  cannot  be 
altered  to  eliminate  ESOH  hazards,  the  Program  Manager  must  identify  the  risks, 
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prioritize  the  risks  by  severity,  determine  how  to  mitigate  the  risks,  and  formally  accept 
any  risks  that  could  not  be  eliminated.  The  Program  Manager  should  also  define 
allowable  levels  of  risk  in  accordance  with  standards  set  forth  in  MIL-STD-882D  and 
other  safety  regulations.  In  all  instances,  the  Program  Manager  must  document  any  risks 
that  were  accepted  throughout  the  lifecycle  and  any  hazardous  materials  with  which 
operators  and  maintainers  may  come  into  contact  (Air  Force  Human  Systems  Integration 
Office,  2003;  Malone  and  Miller,  2003). 

Areas  of  consideration  may  include,  but  are  not  limited  to:  threats  from  acoustic 
energy;  biological  and  chemical  substances;  atmospheric  hazards;  ionizing  and  non¬ 
ionizing  radiation;  fire  and  explosions;  shock;  pressure  extremes;  temperature  extremes 
and  humidity;  trauma;  vibration,  and  uncontrolled  mechanical,  electrical  or  fluid  energy 
releases  (Malone  and  Miller,  2003;  Department  of  Defense,  2004). 

3.  Specific  Objectives  of  an  HSI  Environment,  Safety  and  Occupational 
Health  (ESOH)  Evaluation 

•  Eliminate  or  minimize  injury,  death,  and  lost  work  hours  of  system  operators, 
maintainers,  and  supporters  caused  by  instantaneous,  cumulative  and  residual  effects 
of  system  use  (Malone  and  Miller,  2003). 

•  Develop  a  system  safety  plan  (Malone  and  Miller,  2003). 

•  Review  preliminary  engineering  designs  for  safety  limitations  (Malone  and  Miller, 
2003). 

•  Eliminate  or  minimize  safety  risks  associated  with  the  system  through  an  iterative 
review  process  (Malone  and  Miller,  2003). 

•  Document  safety  requirements  to  be  implemented  in  system  specification/design 
(Malone  and  Miller,  2003). 

•  Conduct  a  HSI/DOTMLPF  trade-off  analysis  to  ensure  safety  requirements  do  not 
invalidate  assumptions  made  in  other  domains  (Malone  and  Miller,  2003). 

•  Complete  a  safety  and  health  analysis  to  identify  unacceptable  risks  associated  with 
the  system  design  and  planned  implementation  (Malone  and  Miller,  2003).  . 

•  Identify  any  special  tests  required  to  demonstrate  or  otherwise  verify  the  proper 
functioning  of  safety  designs  and  equipment  (Malone  and  Miller,  2003). 

•  Collect  data  on  safety  testing,  failure  analysis,  mishap  reports,  and  other  system  tests 
with  safety  implications  for  analysis  and  decision  support  (Malone  and  Miller,  2003). 

•  Ensure  required  warnings,  cautions,  and  special  safety  procedures  are  documented, 
posted,  and  supported  with  training  (Malone  and  Miller,  2003). 
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•  Test  safety  and  warning  devices,  life  support  equipment,  and  personal  protective 
equipment  for  effectiveness  in  all  anticipated  operating  environments  (Malone  and 
Miller,  2003). 

4.  A  Practical  Environment,  Safety  and  Occupational  Health  (ESOH) 
Evaluation  for  REF  Systems 

The  REF  is  procuring  many  of  its  products  from  the  commercial  sector,  and  it  is 
likely  that  the  product  manufacturers  have  conducted  safety  analyses  during  product 
development  in  order  to  avoid  the  consequences  of  a  liability  lawsuit.  However,  but  the 
REF  analyst  should  not  simply  accept  this  assumption  without  verifying  that  testing  was 
conducted  and  the  reported  results  are  valid.  Even  if  testing  was  conducted  by  the 
manufacturer,  the  REF  analyst  must  verify  that  the  results  of  the  testing  can  be 
generalized  to  the  system’s  military  concept  of  operations. 

To  assist  with  this  determination,  the  REF  analyst  is  supported  by  the  Army  Test 
and  Evaluation  Command  (ATEC).  For  new  REF  products,  ATEC  will  provide  an 
objective  Safety  Certification  for  the  system.  For  REF  systems  with  repercussions  on 
survivability  and  lethality,  ATEC  will  include  a  Capabilities  and  Limitations  Report  with 
the  Safety  Certification  (Defense  Acquisition  University,  2006).  While  the  ATEC  Safety 
Certification  is  beneficial  to  the  REF  process,  evaluations  of  complex  systems  may  be 
time  consuming  and  significantly  delay  delivery  of  the  equipment  to  the  field. 

The  REF  analyst  may  be  able  to  facilitate  the  ATEC  analysis  by  providing  an 
initial  subjective  assessment  of  the  system  to  identify  critical  ESOH  risks.  The  goals  of 
this  analysis  would  be  to  eliminate  products  with  gross  safety  concerns  and  provide  a 
focus  for  further  objective  safety  evaluations.  The  overarching  subjective  evaluation 
might  use  a  system  similar  to  the  risk  management  matrix  in  which  the  severity  of  the 
consequence  is  compared  to  the  probability  of  occurrence  (Clark  and  Goulder,  2002).  As 
with  all  other  domains,  to  standardize  measures  of  severity,  the  REF  analyst  should  use 
the  following  delineation:  “[Category  1]  -  Catastrophic  -  may  cause  death  or  system  loss; 
[Category  2]  -  Critical  -  may  cause  severe  bodily  injury,  severe,  occupational  illness,  or 
major  system  damage;  [Category  3]  -  Marginal  -  may  cause  minor  bodily  injury,  minor 
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occupational  illness,  or  minor  system  damage;  and  [Category  4]  -  Negligible  -  may  cause 
less  than  minor  bodily  injury,  occupational  illness,  or  minor  system  damage”  (Clark  and 
Goulder,  2002  p.  90). 

If  a  deficiency  is  noted  through  any  method  of  evaluation,  the  REF  analyst  should 
make  recommendations  to  address  the  deficiency  through  system  design,  training,  or 
other  protective  measures  (Malone  and  Miller,  2003).  Any  decision  to  accept  potential 
risks  should  be  formally  documented  by  the  REF  analyst  and  forwarded  to  the  decision 
maker  for  consideration  (Naval  Sea  Systems  Command,  2005). 

This  discussion  of  the  ESOH  domain  suggests  several  questions  that  the  REF 
analyst  could  pose  to  evaluate  the  ESOH  domain  (refer  to  the  ESOH  portion  of  appendix 
A  for  questions). 
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IX.  HUMAN  FACTORS  DOMAIN 


A.  HUMAN  FACTORS  DEFINED 

Human  Factor  Engineering  (HFE)  is  a  multidisciplinary  domain,  which  includes 
topics  that  encompass  all  aspects  of  human-system  to  human  interactions.  The  factors 
associated  with  HFE  are  (Critical  Process  Assessment  Tool,  1998): 

•  “Anthropometries  Factor  -  Human  Physical  Dimensions,  Body  Posture,  Repetitive 
Motion,  Physical  Interface.” 

•  “Sensory  Factors  -  Hearing,  Vision,  Touch,  Balance.” 

•  “Cognitive  Factors  -  Mental  Ability,  Skills,  Decision  Making,  Training 
Requirements.” 

•  “Psychological  Factors  -  Human  Needs,  Attitudes,  Expectations,  Motivations.” 

•  “Physiological  Factors  -  Human  Reactions  to  Environments,  Strength  (lifts,  grip, 
carrying,  etc.),  Endurance.” 

According  to  the  Handbook  of  Human  Factors,  “Human  Factors  Engineering 
(HFE)  and  applied  ergonomics  are  concerned  with  the  application  of  the  data  and 
principals  of  human  factors  and  ergonomics  to  the  design  of  equipment,  subsystems,  and 
systems”  (Christensen,  1987,  p.  8).  This  perspective  implies  that  human  factors 
engineering  and  ergonomics  are  synonymous  terms  and  the  focus  is  the  overall 
improvement  of  systems  and  working  conditions  between  the  human  and  the  system. 
This  definition  serves  as  the  foundation  on  which  the  DoD  has  based  its  HFE  instruction. 

AR  602-1  is  the  regulation  governing  HFE  within  the  Army.  This  regulation 
defines  HFE  “as  a  comprehensive  technical  effort  to  integrate  all  personnel 
characteristics  (skills,  training  implications,  behavioral  reactions,  human  performance, 
anthropometric  data  and  biomedical  factors)  into  Army  doctrine  and  systems  to  assure 
operational  effectiveness,  safety,  and  freedom  from  health  hazards”  (AR  602-1,  1986,  p. 
1). 

B.  HUMAN  FACTORS  DOMAIN  IN  HSI 

The  human  factors  domain  in  HSI  has  adopted  the  definition  used  by  the  Army 
MANPRINT  program.  It  supplements  that  AR  602-1  by  emphasizing  the  integration 
aspect.  Specifically,  this  definition  addresses  HFE  as  “the  integration  of  human 

65 


characteristics  into  system  definition,  design,  development  and  evaluation  to  optimize 
human-machine  performance  under  operational  conditions” 

(www.manprint.anny.mil/manprint/index.htm).  The  HSI  practitioner’s  primary  concerns 
in  relation  to  the  human  factors  domain  is  to  ensure  the  program  manager  considers  HFE 
during  the  early  stages  of  the  system  lifecycle  and  to  ensure  that  HFE  issues  are 
considered  throughout  the  lifecycle  of  the  program.  The  benefit  of  this  process  is  that 
potential  cost-effective  measures  are  taken  and  that  “system  designs  will  minimize  or 
eliminate  system  characteristics  that  require  excessive  cognitive,  physical,  or  sensory 
skills;  entail  extensive  training  or  workload-intensive  tasks;  result  in  mission-critical 
errors;  or  produce  safety  or  health  hazards”  (Department  of  Defense,  2003). 

C.  HSI,  HUMAN  FACTORS  AND  THE  REF 

The  challenges  presented  to  the  REF  within  the  human  factors  domain  are 
twofold.  The  first  challenge  the  REF  encounters  is  the  alignment  between  the  product’s 
specifications  and  the  user  population.  Within  the  traditional  acquisition  approach  to 
system  development,  products  are  customized  for  the  specific  user  population.  However, 
the  types  of  products  the  REF  acquires  are  not  constrained  to  government  or  military  user 
populations.  Given  that  the  products  the  REF  deploys  are  generally  NDIs/  COTS  and  are 
normally  designed  for  the  general  public,  there  is  no  guarantee  that  these  products  can  be 
fielded  for  operational  use  and  are  designed  for  the  typical  military  Soldier  user. 

The  second  challenge  is  the  manner  in  which  HFE  interacts  with  other  domains. 
For  example,  if  a  product  forces  a  Soldier  to  be  cognitively  overloaded  while  on  a 
dangerous  mission,  it  could  lead  to  a  severe  decrease  in  the  Soldier’s  survivability. 
Therefore,  the  REF  analyst  must  consider  the  impact  HFE  issues  (i.e.,  psychological, 
cognitive,  anthropometries,  sensory,  and  physiological)  on  other  domains  and  how  other 
domains  on  HFE. 

1.  Goal  of  an  HSI  Human  Factors  Engineering  Evaluation 

One  of  the  goals  of  a  Human  Factors  Engineering  evaluation  is  to  incorporate  the 
capabilities  and  limitations  of  the  operators,  maintainers,  and  supporters  into  the 
definition,  design  and  development  of  systems  to  maximize  mission  effectiveness  while 
ensuring  the  system  meets  performance  criteria  for  safety,  habitability,  and  survivability 
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(Department  of  Defense,  2003).  When  employed  effectively,  Human  Factors 
Engineering  optimizes  the  mix  of  manpower,  personnel  and  training  by  minimizing  or 
eliminating  system  characteristics  that  require  unusual  or  unusually  high  levels  of 
knowledge,  skills,  or  cognitive,  sensory  or  physical  abilities.  Ineffective  employment  of 
HFE  may  result  in  excessive  workload  or  require  excessive  training  to  prevent  mission 
critical  errors  that  would  lead  to  unacceptable  levels  of  injury  or  death  (Department  of 
Defense,  2003;  Department  of  Defense,  2004).  Unfortunately,  variances  in  the  military 
population  mean  that  no  system  can  be  designed  to  fully  accommodate  all  users.  The 
challenge  in  HFE  is  to  design  system’s  to  accommodate  as  large  a  portion  of  the 
population  as  possible. 

Human  Factors  Engineering  examines  human-system  interaction,  including: 
human-computer  interfaces,  human-machine  design,  system  software,  communications, 
maintenance,  procedures,  and  working  environment  (Malone  and  Miller,  2003;  HSI 
VIRTUAL  SYSCOM  WORKING  GROUP,  2005).  Human  Factors  Engineering  seeks  to 
develop  improved  interfaces  to  achieve  required  levels  of  performance  effectiveness 
during  operations,  maintenance,  and  support  activities  thereby  allowing  decision  makers 
to  make  economically  sound  decisions  concerning  manpower,  personnel,  training  and 
cost  (Malone  and  Miller,  2003).  Generally,  Human  Factors  Engineering  also  considers 
system  failures  that  lead  to  (or  have  the  potential  to  lead  to)  catastrophic  incident. 
(Wickens  and  Lee,  2003). 

2.  Human  Factors  Engineering  Evaluation  in  a  Traditional  Acquisition 
Setting 

DoD  Instruction  5000.2  directs  the  program  manager  to  “take  steps  (e.g.,  contract 
deliverables  and  Government/contractor  IPT  teams)  to  ensure  human  factors 
engineering/cognitive  engineering  is  employed  during  systems  engineering  over  the  life 
of  the  program  to  provide  for  effective  human-machine  interfaces  and  to  meet  HSI 
requirements”  (Department  of  Defense,  2003).  Work  in  Human  Factors  Engineering 
should  begin  during  the  Concept  Refinement  and  Technology  Development  phases  of  the 
acquisition  lifecycle  to  ensure  that  the  concepts  for  system  performance  do  not  exceed  the 
capabilities  of  the  target  audience  (Malone  and  Miller,  2003).  In  these  early  acquisition 
phases,  Human  Factors  Engineering  analysts  evaluate  legacy  systems  and  expected 
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operator  tasks  to  determine  user  needs  and  identify  and  parse  required  functions,  tasks, 
and  workload  using  Top  Down  Requirement  Analysis  (TDRA)  and  to  other 
methodologies  as  described  in  MIL-HDBK-46855A  (Department  of  Defense,  2004; 
Office  of  the  Deputy  Chief  of  Staff  Gl,  2005). 

The  TDRA  should  begin  with  a  baseline  scenario  used  to  develop  a  mission  and 
to  define  required  functions  for  accomplishment  of  mission  objectives.  Function  should 
be  broken  down  into  the  tasks  required  to  accomplish  each  function.  Finally,  each  task 
should  be  analyzed  to  define  required  human  performance  parameters  given  the 
envisioned/tactical  environment  and  system  capabilities  (Department  of  Defense,  2004). 

Human  Factors  Engineering  requirements  are  normally  stated  in  terms  of  the  user 
population  as  it  relates  to  the  cognitive,  sensory,  or  physical  knowledge,  skills,  or 
abilities.  Cognitive  requirements  are  normally  stated  as  response  times  or  measures  of 
accuracy  and  reliability.  Physical  requirements  are  stated  as  either  anthropometric, 
strength  or  weight  standards.  Sensory  requirements  are  normally  stated  as  visual, 
auditory,  or  olfactory  standards  (Department  of  Defense,  2004). 

As  the  system  moves  from  Concept  Refinement  and  Technology  Development  to 
System  Development  and  Demonstration  phase  of  the  acquisition  lifecycle,  HFE  analysts 
become  more  involved  with  testing  design  alternatives  using  mock-ups,  modeling  and 
simulation,  and  system  prototypes  (Office  of  the  Deputy  Chief  of  Staff  Gl,  2005). 
During  this  phase  of  the  acquisition  lifecycle,  the  HFE  analysts  should  specify 
requirements  for  human-machine  and  human-computer  interface  design  by  addressing  the 
eight  classes  of  interfaces  -  functional,  informational,  environmental,  cooperational, 
organizational,  operational,  cognitive,  and  physical  -  required  for  effective  human 
performance  (Malone  and  Miller,  2003;  Department  of  Defense,  2004).  HFE  tools  such 
as  the  NASA  Task  Load  Index  (NASA  TLX),  IMPRINT,  and  JACK  (human  figure 
modeling  software)  assist  the  analyst  in  identifying  KSA  thresholds  for  the  target 
population  to  be  used  in  planning  and  later  testing  evolutions  (Department  of  Defense, 
2004). 

Once  the  system  moves  to  the  Low  Rate  Initial  Production  phase  and  into 
Operational  Test  and  Evaluation,  the  goal  of  the  HFE  analyst  is  to  confirm  that  the 
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system  design  meets  the  Human  Factors  requirements  defined  in  earlier  acquisition 
phases.  Additionally,  the  HFE  analyst  should  attempt  to  identify  new  design  or 
procedural  issues  that  will  have  an  adverse  effect  on  the  human  in  the  system  (Office  of 
the  Deputy  Chief  of  Staff  Gl,  2005).  If  any  such  issues  exist,  the  Human  Systems 
Integration  professional  should  detennine  if  the  deficiency  can  be  sufficiently  addressed 
through  one  of  the  other  HSI  domains.  If  no  reasonable  solution  can  be  found  through 
HSI  domain  trade-offs,  then  the  system  must  revert  to  previous  stages  of  the  acquisition 
lifecycle  to  correct  these  deficiencies. 

3.  Specific  Objectives  of  an  HSI  Human  Factors  Engineering  Evaluation 

•  Influence  system  definition,  design,  and  development  by  placing  the  capabilities  and 
limitations  of  the  target  audience  at  the  forefront  of  decision  maker  considerations 
(Department  of  Defense  2003;  Malone  and  Miller  2003). 

•  Maximize  the  effectiveness  of  new  systems  by  reducing  required  knowledge,  skills 
and  abilities;  sensory,  cognitive  and  physical  workload;  training;  and  mission  critical 
error  rates  (Department  of  Defense,  2003;  Malone  and  Miller,  2003;  Department  of 
Defense,  2004). 

•  Optimize  the  number  of  personnel  required  to  safely  operate  the  new  system 
(Department  of  Defense,  2003). 

•  Ensure  that  system  performance  standards  are  not  considered  independently  of 
humans  in  the  loop  (Office  of  the  Deputy  Chief  of  Staff  Gl,  2005). 

•  Determine  key  task  and  function  requirements  through  a  top  down  requirements 
analysis  (Malone  and  Miller,  2003;  Department  of  Defense,  2004). 

•  Reduce  the  time  required  to  perform  a  task  through  increased  reliability,  accuracy, 
and  ability  to  correct  for  errors  (Malone  and  Miller,  2003). 

•  Optimize  human-machine  interfaces  to  improve  safety,  usability,  reliability, 
survivability  and  time  to  respond  (Malone  and  Miller,  2003). 

•  Influence  workspace  design  and  environment  to  improve  habitability  and  maximize 
the  number  of  people  in  the  user  population  (Malone  and  Miller,  2003). 

•  Ensure  technology  and  automation  implementations  do  not  degrade  situational 
awareness  by  causing  cognitive  or  sensory  overload  (Malone  and  Miller  2003). 

•  Ensure  system  design  does  not  unduly  impact  timeliness  and  ability  to  conduct 
required  maintenance  tasks  (Malone  and  Miller,  2003). 

•  Verify  that  the  target  audience  is  required  to  correctly  operate  the  system  to  meet 
stated  perfonnance  standards  in  all  operational  environments  and  conditions, 
including  donning  of  personal  protective  equipment  (Malone  and  Miller,  2003). 
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4.  A  Practical  Human  Factors  Engineering  Evaluation  for  REF  Systems 

Because  the  REF  is  primarily  involved  with  NDI,  COTS,  and  GOTS  items,  the 
ability  of  analysts  to  significantly  influence  the  design  of  new  products  is  limited.  When 
conducting  a  Human  Factors  Engineering  evaluation,  the  REF  analysts  typically  find 
themselves  in  one  of  three  possible  situations:  (1)  evaluating  a  new  sole  source  project, 
(2)  evaluating  a  new  product  with  multiple  sourcing  possibilities,  or  (3)  re-evaluating  a 
previous  REF  project  prior  to  returning  it  to  the  field.  Each  of  these  situations  presents  a 
unique  way  for  the  REF  to  influence  decisions  involving  product  design  and 
employment. 

Clearly,  the  most  restrictive  of  the  situations  above  is  when  the  REF  analyst  is 
evaluating  a  new  sole  source  project  because  he  or  she  is  limited  by  time  constraints  to 
the  product  the  contractor  has  to  offer.  In  this  situation,  the  REF  analyst  cannot  have 
significant  influence  over  the  design  of  the  project,  and  must  make  compensation  for 
human  factors  concerns  through  the  interaction  with  other  domains.  In  order  to  make  this 
happen,  the  REF  must  conduct  an  abbreviated  TDRA  to  identify  key  functions  and  tasks 
the  system  must  provide.  With  the  key  functions  and  tasks  identified,  the  REF  analyst 
can  use  the  new  system  together  with  inputs  from  the  Target  Audience  Description  to 
accomplish  these  functions  and  tasks  in  order  to  begin  making  determinations  on 
manpower,  personnel  and  training  levels  required  to  meet  the  desired  perfonnance 
standard. 

As  an  example,  if  the  REF  is  evaluating  a  one-of-a-kind,  low  technology,  300- 
pound,  man-portable  piece  of  equipment,  the  Target  Audience  Description  may  imply 
that  the  system  requires  a  three  man  heavy  lift.  In  order  to  use  this  piece  of  equipment 
effectively,  the  system  will  require  three  dedicated  people  with  specific  biomechanical 
capabilities  to  move  the  system.  This  determination  will  affect  the  manpower  and 
personnel  domains,  may  have  an  impact  on  safety  and  survivability,  and  might  convince 
the  decision  maker  that  the  system  is  not  feasible  for  the  given  concept  of  operations. 

In  the  second  possible  scenario,  a  REF  project  may  have  multiple  product  sources. 
This  situation  is  a  little  less  restrictive  to  the  REF  analyst  because  while  there  is  still  no 
ability  to  make  specific  engineering  design  changes,  the  REF  analyst  might  be  able  to 
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choose  between  alternatives  from  a  user-centered  perspective.  In  this  situation,  an 
abbreviated  TDRA  is  conducted  to  develop  key  tasks  and  functions,  and  each  alternative 
is  evaluated  using  these  key  tasks  and  functions  and  the  Target  Audience  Description. 
For  example,  if  two  products  are  being  evaluated,  a  medium  technology,  200  pound, 
man-portable  piece  of  equipment  and  a  high  technology,  100  pound,  man-portable  piece 
of  equipment,  then  the  REF  analyst  can  use  applicable  Human  Factors  standards  to 
determine  which  piece  of  equipment  best  addresses  the  key  functions  and  tasks.  The  true 
challenge  of  this  situation  is  prioritizing  the  system  requirements  in  order  to  optimize  the 
trade-offs  between  HSI  domains. 

In  the  third  possible  scenario,  the  REF  decides  to  re-offer  a  previously  evaluated 
piece  of  equipment  to  other  units  in  the  field,  using  a  spiral  development  mentality  to 
offer  performance  improvements  with  each  increment.  The  REF  is  able  to  offer  specific 
design  recommendations  to  be  acted  upon  by  the  equipment  designer.  Although  rare,  this 
situation  presents  the  best  opportunity  for  the  REF  to  tailor  products  to  meet  the 
requirements  of  Soldiers  in  the  field. 

As  an  example  of  this  concept,  consider  a  laser  targeting  system.  In  the  first 
iteration,  the  REF  analyst  will  be  in  situation  one  or  two,  where  no  input  on  system 
design  is  possible.  If  the  product  meets  with  immediate  success  if  the  field,  the  REF  can 
use  lessons  learned  to  correct  product  deficiencies  before  sending  subsequent  iterations  to 
the  field.  If  the  lessons  learned  can  be  gathered  early  enough  then  the  time  constraints 
that  normally  hamper  design  changes  in  such  situations  maybe  overcome. 

This  discussion  of  the  HFE  domain  suggests  several  questions  that  the  REF 
analyst  could  pose  to  evaluate  the  HFE  domain  (refer  to  the  HFE  portion  of  appendix  A 
for  questions). 
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X.  THE  DATABASE 


A.  ABOUT  THE  HUMAN  SYSTEMS  INTEGRATION  (HSI)  ANALYSIS 

TOOL 

The  goal  of  ABRAHAM  is  to  guide  REF  analysts  and  decision  makers  through 
the  domains  of  HSI  to  determine  where  new  products  present  unacceptable  risks  to 
operators,  maintainers  and  support  personnel.  ABRAHAM  provides  up  front,  time 
sensitive,  and  mission  oriented  subjective  analysis  in  the  absence  of  objective  data  from 
product  manufacturers  and  other  organizations.  Once  a  product  has  been  fielded, 
ABRAHAM  creates  customizable  surveys  to  elicit  responses  from  Soldiers  in  the  field 
about  the  suitability  of  the  product. 

ABRAHAM  was  implemented  using  Microsoft  Access  database  software,  which 
is  a  part  of  the  Microsoft  Office  Professional  software  package.  ABRAHAM  makes 
extensive  use  of  queries  to  convert  user  input  into  a  tailored  output  of  HSI  domain 
questions  for  use  in  alternatives  analysis  and  product  evaluation.  ABRAHAM  also 
provides  basic  analysis  of  questionnaire  responses  and  a  written  report,  which  addresses 
areas  of  concern  for  follow-on  analysis  by  REF  analysts  and  contracted  Subject  Matter 
Experts. 

1.  HSI  Analysis  Tool  Structure 

ABRAHAM’s  structure  is  depicted  below  (Figure  6).  There  are  two  main 
sections  to  ABRAHAM,  the  Alternatives  Analysis  (AoA)  section  and  the  Product 
Evaluation  (PE)  section.  Each  area  of  the  tool  is  described  with  screenshots  later  in  this 
chapter.  The  entire  HSI  Analysis  Tool  is  designed  to  allow  operation  with  minimal  use 
of  the  Microsoft  Access  menu  system. 
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Figure  6.  HSI  Analysis  Tool  Structure 


2.  HSI  Analysis  Tool  Installation 

To  install  ABRAHAM,  simply  copy  the  HSI  Questions  Database.MDB  file  from 
the  CD-ROM  to  any  large  storage  device  on  your  computer  or  network.  ABRAHAM 
will  require  at  least  20  megabytes  of  free  space,  and  users  should  expect  the  fde  size  to 
grow  with  continued  use  of  the  program.  To  keep  the  HSI  Questions  Database.MDB  file 
at  its  minimum  size,  it  is  recommended  that  users  routinely  use  the  “Compact  and 
Repair”  tool  by  going  to  ToolsYDatabase  Utilities\Compact  and  Repair  Database  on  the 
Access  menu  bar. 

3.  Opening  ABRAHAM 

To  open  ABRAHAM,  double-click  the  mouse  with  the  cursor  on  the  HSI 
Questions  Database.MDB  file.  When  the  file  opens,  the  user  may  be  prompted  with  a 
security  warning  that  states  the  file  may  not  be  safe  to  open  if  it  contains  code  intended  to 
hann  your  computer  (Figure  7). 
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Figure  7.  Security  Warning  Page 

This  is  a  standard  Microsoft  warning  when  macro  security  level  is  set  to  medium 
and  Access  cannot  verify  that  the  file  was  created  by  a  trusted  source.  For  more 
information  on  this  security  warning,  please  use  your  Internet  browser  to  go  to  the 
following  link:  http://office.microsoft.com/en-us/access/HA01 122598 1033 .aspx#070. 

To  eliminate  this  warning  on  your  computer,  please  use  your  Internet  browser  to 
go  to  the  following  link: 

http://office.microsoft.com/en-us/access/HA01 122598 1033  .aspx#200. 

B.  USING  ABRAHAM 

1.  The  Welcome  Screen 

Once  the  database  file  is  opened,  the  user  will  be  greeted  with  a  welcome  screen 
similar  to  the  one  depicted  below  (Figure  8).  This  Welcome  screen  will  allow  the  user  to 
enter  the  Alternatives  Analysis  section,  the  Product  Evaluation  section,  or  exit 
ABRAHAM.  The  user  can  use  the  Product  Evaluation  section  independently  of  the 
Alternatives  Analysis  Section.  By  clicking  the  Conduct  Alternatives  Analysis  button,  the 
Alternatives  Analysis  Options  page  will  open. 


75 


||j*l  Microsoft  Access  -  [Questions] 

- 

:  il  File  Edit  View  Insert  Format 

Records  Tools  Window  Help 

Type  a  question  for  help  »  .  5  x 

Ji5l» 

1  utJ  S|  .LL(  ■;  .  1  1 

•  |  MS  Sans  Serif 

-  |  8  -  B  /  U  |  E 

-|A.|  -1  •[ 

► 

NAVAL 

POSTGRADUATE 

SCHOOL 

HSI  Evaluation  Tool 


Would  you  like  to: 


Conduct 

Alternatives  Analysis; 

Conduct 

Product  Evaluation 

Exit  HSI  Evaluation  Tool 

Record:  [  H  |  i  |  T  ►  [  H  |  >  -  of  1 

Form  View 


Figure  8.  Welcome  Screen 


2.  The  Conduct  Alternatives  Analysis  Options  Page 

The  Alternatives  Analysis  Options  page  (Figure  9)  will  allow  the  user  to  Enter 
New  Questions,  Select  Questions  for  Evaluation,  Conduct  the  Alternatives  Analysis,  or 
Return  to  the  Welcome  Screen.  The  Select  Questions  for  Evaluation  button  can  be  used 
independently  of  the  Enter  New  Questions  button,  but  the  Conduct  Alternatives  Analysis 
is  dependent  on  the  entries  made  in  the  Select  Questions  for  Evaluation  page.  By 
clicking  the  Enter  New  Questions  button,  the  user  will  be  taken  to  the  Alternatives 
Analysis  Enter  New  Questions  page. 
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Figure  9.  Alternatives  Analysis  Options  Page 

3.  The  Enter  New  Questions  Page 

The  Enter  New  Questions  Page  (Figure  10)  provides  users  to  the  ability  add 
questions  to  the  database.  Users  are  not  required  to  utilize  this  page,  but  this  form  may 
be  useful  if  the  user  is  examining  a  unique  piece  of  equipment  that  is  not  addressed  by  the 
pre-loaded  questions.  The  Enter  New  Questions  is  by  far  the  most  complicated  form  of 
ABRAHAM. 

At  the  top  of  the  Enter  New  Questions  page,  users  will  find  a  field  for  Question 
Key  is  automatically  filled  in  by  the  database  tool.  The  question  key  will  not  necessarily 
be  in  sequential  order  and  will  not  match  the  record  number  shown  on  the  bottom  left 
side  of  the  screen.  The  Question  Key  field  is  locked;  users  will  not  be  allowed  to  change 
the  question  key. 

The  next  field  is  the  Question  field,  this  is  the  field  that  the  user  will  use  to  enter 
the  new  question.  The  Question  field  can  be  used  to  ask  Likert  scale,  yes/no,  and  open- 
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ended  questions.  Users  should  remember  to  keep  their  questions  as  general  as  possible  so 
that  they  can  be  applied  to  multiple  system  types. 


To  the  right  of  the  Question  Field,  the  user  will  find  two  drop-down  boxes.  The 
first  drop-down  box  allows  the  user  to  assign  an  HSI  or  DOTMLPF  domain  to  the 
question.  This  field  is  not  required,  but  it  is  useful  because  the  questions  will  be  grouped 
by  these  domains  in  later  reports.  The  next  drop-down  box  is  use  to  assign  the  level  of 
importance  to  the  question.  Users  will  have  the  option  of  assigning  a  value  of  high, 
medium,  or  low,  and  the  value  will  be  converted  to  a  numerical  score  (high=3, 
medium=2,  low=l)  to  be  used  as  a  multiplier  to  determine  a  final  score  for  the  system. 
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Figure  10.  The  Enter  New  Questions  Page 


Below  the  question  field,  the  user  will  find  two  check  boxes  on  the  left  side  of  the 
screen.  The  first  check  box  asks  the  user  if  the  question  requires  a  Likert  response.  If 
unchecked  then  all  fields  below  this  question  will  be  hidden.  The  second  check  box  asks 
the  user  if  a  not  applicable  response  is  required  for  this  question.  If  unchecked  then  the 
Likert  Response  N/A  field  will  be  hidden. 
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The  Likert  Responses  are  designed  to  support  five  responses.  Likert  Response  1 
should  always  contain  the  worst  alternative,  while  Likert  Response  5  should  contain  the 
most  favorable  alternative.  Using  this  format  is  critical  to  the  program’s  logic,  and  the 
Alternative  Analysis  report  will  be  flawed  if  the  format  is  violated.  When  asking  a 
yes/no  type  question,  the  negative  response  should  be  entered  in  Likert  Response  1  and 
the  positive  response  should  be  entered  in  Likert  Response  5.  For  example,  if  the 
question  asks  “Is  the  system  capable  of  causing  serious  bodily  harm  to  the  user?”  then 
Likert  Response  1  should  be  “Yes”  and  Likert  Response  5  should  be  “No”.  Conversely, 
if  the  question  asks  “Does  the  system  prevent  targeting  friendly  units?”  then  Likert 
Response  1  should  be  “No”  and  Likert  Response  5  should  be  “Yes”. 

Question  scoring  in  the  Alternatives  Analysis  Report  is  based  on  the  Likert 
Response  selected.  Likert  Response  1  is  assigned  1  point,  and  Likert  Response  5  is 
assigned  5  points.  The  Likert  Response  score  is  multiplied  by  the  Importance  multiplier 
to  detennine  the  maximum  possible  score  for  the  question  and  the  score  earned  for  the 
question.  For  example,  if  a  High  importance  question  is  answered  with  Likert  Response 
3  then  the  maximum  score  for  the  question  will  be  1 5  (Importance  =  3x5  points  for  best 
alternative)  and  the  question  score  will  be  9  (Importance  =  3x3  points  for  Likert 
Response  3).  Likert  Response  N/A  is  given  a  score  of  zero  and  the  maximum  possible 
score  will  be  assigned  a  value  of  zero. 

In  the  center  of  the  screen,  the  user  is  presented  with  two  check  boxes  that  ask  if 
the  question  should  be  applied  to  the  Alternatives  Analysis  Questionnaire  or  the  Product 
Evaluation  Questionnaire.  One  or  both  of  these  check  boxes  must  be  checked  in  order  for 
the  question  to  be  included  in  later  parts  of  the  tool.  If  neither  of  these  check  boxes  is 
selected  then  the  question  will  not  meet  the  criteria  for  inclusion  the  surveys  generated  by 
the  tool. 

Below  these  two  check  boxes,  the  user  will  find  two  buttons  that  are  used  to 
assign  the  question  to  certain  product  categories.  If  the  user  presses  the  Select  Individual 
Categories  button  then  the  following  screen  will  be  opened  (Figure  11).  If  the  user 
presses  the  Select  All  Categories  button  then  all  the  check  boxes  on  the  following  page 
will  be  selected. 
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Figure  1 1 .  Product  Categories  Page 


The  user  should  select  one  or  more  categories  by  placing  a  check  mark  in  the 
appropriate  boxes.  The  buttons  at  the  bottom  of  the  screen  allow  the  user  to  check  or 
uncheck  all  categories,  and  the  Close  Form  button  will  close  the  form  and  return  the  user 
to  the  Question  Entry  screen  (Figure  10). 

The  final  fields  on  the  Question  Entry  fonn  prompt  the  user  for  Question  Source 
and  Notes/Issues  on  the  Question.  The  Question  Source  field  is  used  to  document  the 
academic  source  of  the  question  or  the  user  who  entered  the  question.  The  Notes/Issues 
on  the  Question  field  is  an  unlimited  memo  field  used  to  document  changes  to  the 
question  or  other  issues  considered  during  the  life  of  the  question. 

On  the  bottom  right  of  the  page,  the  user  will  find  buttons  that  allow  for 
navigation  through  the  question  set.  The  button  on  the  far  left  will  return  the  first 
question  and  the  next  button  will  go  backward  one  question.  The  button  with  the 
binocular  image  will  open  a  search  dialog  box  that  will  search  the  database  based  on  the 
Question  Key  or  Question  field. 
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Figure  12.  Find  and  Replace  Screen 

The  button  to  the  right  of  the  Search  button  will  advance  the  database  by  one 
question,  and  the  next  button  will  advance  the  database  to  the  last  question.  The  button 
with  the  image  of  a  door  will  close  the  page  and  return  the  user  to  the  Alternatives 
Analysis  Options  page  (Figure  9). 

4.  The  Select  Questions  for  Evaluation  Page 

The  second  option  on  the  Alternatives  Analysis  Options  page  is  the  Select 
Questions  for  Evaluation  button.  When  the  user  selects  this  option,  the  following  screen 
is  opened  (Figure  13).  On  this  form,  the  user  is  presented  with  thirty  systems  categories 
and  asked  to  select  those  categories  that  apply  to  the  system  being  evaluated.  As  an 
example,  if  the  user  were  evaluating  a  portable  GPS  system  then  the  user  might  select 
Navigation  Equipment,  Computing  and  Software,  and  Batteries  and  Power  Supplies. 

On  the  bottom  of  the  screen,  there  are  two  buttons  for  user  input.  Once  the  user 
has  selected  all  applicable  categories,  pressing  the  Get  the  Questions  button  will  allow  the 
database  to  select  the  questions  and  return  the  user  to  the  Alternatives  Analysis  Options 
Page  (Figure  9).  Pressing  the  Close  Form  button  will  close  the  fonn  and  return  the  user 
to  the  Alternatives  Analysis  Options  page  (Figure  9)  without  selecting  any  questions. 
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Figure  13.  Evaluation  Tool  Alternatives  Page 

5.  The  Conduct  Alternatives  Analysis  Page 

When  the  user  selects  the  Conduct  Alternatives  Analysis  button  on  the 
Alternatives  Analysis  Options  page  (Figure  9),  the  following  screen  will  open  (Figure 
14).  The  questions  on  this  page  are  based  on  the  work  done  by  the  user  on  the  Select 
Questions  for  Evaluation  page.  If  the  user  skips  the  Select  Questions  for  Evaluation  page 
and  opens  this  page  directly  then  the  program  will  generate  a  survey  based  on  the  last 
user’s  input  on  the  Select  Questions  for  Evaluation  page. 

When  the  Conduct  Alternatives  Analysis  page  opens  (Figure  14),  the  user  will  be 
presented  with  a  survey  consisting  of  the  questions  in  the  database  that  are  assigned  to  the 
Alternatives  Analysis  and  meet  the  criteria  established  in  the  Select  Questions  for 
Evaluation  page.  The  user  should  answer  all  questions  as  honestly  as  possible. 

When  all  questions  have  been  answered,  the  user  should  press  the  Show  HSI 
Analysis  button  to  conduct  the  analysis  and  show  the  HSI  analysis  report.  Selecting  the 
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Close  Form  button  will  close  the  form  and  return  the  user  to  the  Alternatives  Analysis 
Options  Page  (Figure  9)  without  displaying  a  report. 
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Figure  14.  Alternatives  Analysis  Questionnaire  Page 


When  the  user  presses  the  Show  HSI  Analysis  button  the  following  report  will  be 
generated  in  print  preview  mode  (Figure  15).  Pressing  the  Printer  icon  will  send  the 
report  to  the  default  printer.  Pressing  the  Magnifying  Glass  icon  will  zoom  in  and  out  of 
the  report.  Pressing  the  Close  button  on  the  menu  bar  will  close  the  report  and  return  the 
user  to  the  Conduct  Alternatives  Analysis  page  (Figure  9). 


83 


Figure  15.  Results  Evaluation  Page 
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6. 


The  Conduct  Product  Evaluation  Options  Page 


On  the  Welcome  Screen  (Figure  16)  the  user  can  also  open  the  Product 
Evaluation  Options  page  by  pressing  the  Conduct  Product  Evaluation  button.  The 
following  is  an  example  of  the  Product  Evaluation  Options  page. 
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Figure  16.  Product  Evaluation  Options  Page 

7.  The  Enter  New  Questions  Page 

The  Enter  New  Questions  Page  (Figure  17)  is  an  exact  replica  of  the  Enter  New 
Questions  Page  (Figure  10)  in  the  Alternatives  Analysis  section.  The  Enter  New 
Questions  page  (Figure  17)  is  designed  to  allow  users  to  add  questions  to  the  database  for 
use  in  future  analysis.  Users  are  not  required  to  utilize  this  page,  but  this  fonn  may  be 
useful  if  the  user  is  examining  a  unique  piece  of  equipment  that  is  not  addressed  by  the 
pre-loaded  questions.  The  Enter  New  Questions  is  by  far  the  most  complicated  form  of 
ABRAHAM. 
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At  the  top  of  the  Enter  New  Questions  page,  users  will  find  a  field  for  Question 
Key  is  automatically  filled  in  by  the  database  tool.  The  question  key  will  not  necessarily 
be  in  sequential  order  and  will  not  match  the  record  number  shown  on  the  bottom  left 
side  of  the  screen.  The  Question  Key  field  is  locked  and  users  will  not  be  allowed  to 
change  the  question  key. 

The  next  field  is  the  Question  field,  and  this  is  the  field  that  the  user  will  use  to 
enter  the  new  question.  The  Question  field  can  be  used  to  ask  Likert  scale,  yes/no,  and 
open-ended  questions.  Users  should  remember  to  keep  their  questions  as  general  as 
possible  so  that  they  can  be  applied  to  multiple  system  types. 

To  the  right  of  the  Question  Field,  the  user  will  find  two  drop-down  boxes.  The 
first  drop-down  box  allows  the  user  to  assign  an  HSI  or  DOTMLPF  domain  to  the 
question.  This  field  is  not  required,  but  it  is  useful  because  the  questions  will  be  grouped 
by  these  domains  in  later  reports.  The  next  drop-down  box  is  use  to  assign  the  level  of 
importance  to  the  question.  Users  will  have  the  option  of  assigning  a  value  of  high, 
medium,  or  low,  and  the  value  will  be  converted  to  a  numerical  score  (high=3, 
medium=2,  low=l)  to  be  used  as  a  multiplier  to  determine  a  final  score  for  the  system. 


Microsoft  Access  -  [Questions] 

:  :H  File  Edit  View  Insert  Format  Records  Tools  Window  Help  Type  a  question  for  help  »  _  B  X 


Figure  17.  Enter  New  Questions  Page 
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Below  the  question  field,  the  user  will  find  two  check  boxes  on  the  left  side  of  the 
screen.  The  first  check  box  asks  the  user  if  the  question  requires  a  Likert  response.  If 
unchecked  then  all  fields  below  this  question  will  be  hidden.  The  second  check  box  asks 
the  user  if  a  not  applicable  response  is  required  for  this  question.  If  unchecked  then  the 
Likert  Response  N/A  field  will  be  hidden. 

The  Likert  Responses  are  designed  to  support  five  responses.  Likert  Response  1 
should  always  contain  the  worst  alternative,  while  Likert  Response  5  should  contain  the 
most  favorable  alternative.  Using  this  format  is  critical  to  the  program’s  logic,  and  the 
Alternative  Analysis  report  will  be  flawed  if  the  format  is  violated.  When  asking  a 
yes/no  type  question,  the  negative  response  should  be  entered  in  Likert  Response  1  and 
the  positive  response  should  be  entered  in  Likert  Response  5.  For  example,  if  the 
question  asks  “Is  the  system  capable  of  causing  serious  bodily  harm  to  the  user?”  then 
Likert  Response  1  should  be  “Yes”  and  Likert  Response  5  should  be  “No”.  Conversely, 
if  the  question  asks  “Does  the  system  prevent  targeting  friendly  units?”  then  Likert 
Response  1  should  be  “No”  and  Likert  Response  5  should  be  “Yes”. 

In  the  center  of  the  screen,  the  user  is  presented  with  two  check  boxes  that  ask  if 
the  question  should  be  applied  to  the  Alternatives  Analysis  Questionnaire  or  the  Product 
Evaluation  Questionnaire.  One  or  both  of  these  check  boxes  must  be  checked  in  order  for 
the  question  to  be  included  in  later  parts  of  the  tool.  If  neither  of  these  check  boxes  is 
selected  then  the  question  will  not  meet  the  criteria  for  inclusion  the  surveys  generated  by 
the  tool. 

Below  these  two  check  boxes,  the  user  will  find  two  buttons  that  are  used  to 
assign  the  question  to  certain  product  categories.  If  the  user  presses  the  Select  Individual 
Categories  button  then  the  following  screen  will  be  opened  (Figure  18).  If  the  user 
presses  the  Select  All  Categories  button  then  all  the  check  boxes  on  the  following  page 
will  be  selected. 
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Figure  18.  Product  Categories  Page 


The  user  should  select  one  or  more  categories  by  placing  a  check  mark  in  the  appropriate 
boxes.  The  buttons  at  the  bottom  of  the  screen  allow  the  user  to  check  or  uncheck  all 
categories,  and  the  Close  Form  button  will  close  the  form  and  return  the  user  to  the 
Question  Entry  screen  (Figure  17). 

The  final  fields  on  the  Question  Entry  fonn  prompt  the  user  for  Question  Source 
and  Notes/Issues  on  the  Question.  The  Question  Source  field  is  used  to  document  the 
academic  source  of  the  question  or  the  user  who  entered  the  question.  The  Notes/Issues 
on  the  Question  field  is  an  unlimited  memo  field  used  to  document  changes  to  the 
question  or  other  issues  considered  during  the  life  of  the  question. 

On  the  bottom  right  of  the  page,  the  user  will  find  navigation  buttons.  The  button 
on  the  far  left  will  return  to  the  first  question  and  the  next  button  will  go  backward  one 
question.  The  button  with  the  binocular  image  will  open  a  search  dialog  box  that  will 
search  the  database  based  on  the  Question  Key  or  Question  field. 


88 


Figure  19.  Find  and  Replace  Screen 

The  button  to  the  right  of  the  Search  button  will  advance  the  database  by  one 
question,  and  the  next  button  will  advance  the  database  to  the  last  question.  The  button 
with  the  image  of  a  door  will  close  the  page  and  return  the  user  to  the  Product  Evaluation 
Options  page  (Figure  18). 

8.  The  Select  Questions  for  Evaluation  Page 

The  second  option  on  the  Product  Evaluation  Options  page  is  the  Select  Questions 
for  Evaluation  button.  Again,  this  page  is  an  exact  replica  of  the  Select  Questions  for 
Evaluation  page  in  the  Alternatives  Analysis  section.  When  the  user  selects  this  option, 
the  following  screen  is  opened.  On  this  form,  the  user  is  presented  with  thirty  systems 
categories  and  asked  to  select  those  categories  that  apply  to  the  system  being  evaluated. 
As  an  example,  if  the  user  were  evaluating  a  portable  GPS  system,  then  the  user  might 
select  Navigation  Equipment,  Computing  and  Software,  and  Batteries  and  Power 
Supplies. 

On  the  bottom  of  the  screen,  there  are  two  buttons  for  user  input.  Once  the  user 
has  selected  all  applicable  categories,  pressing  the  Get  the  Questions  button  will  allow  the 
database  to  select  the  questions  and  return  the  user  to  the  Product  Evaluation  Options 
Page.  Pressing  the  Close  Form  button  will  close  the  fonn  and  return  the  user  to  the 
Product  Evaluation  Options  page  without  selecting  any  questions. 
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Figure  20.  Questions  for  Evaluation  Page 


9.  The  Conduct  Product  Evaluation  Page 

When  the  user  selects  the  Conduct  Product  Evaluation  button  on  the  Product 
Evaluation  Options  page  (Figure  16),  the  following  screen  will  open  (Figure  21).  The 
questions  on  this  page  are  based  on  the  work  done  by  the  user  on  the  Select  Questions  for 
Evaluation  page.  If  the  user  skips  the  Select  Questions  for  Evaluation  page  and  opens 
this  page  directly  then  the  program  will  generate  a  survey  based  on  the  last  user’s  input 
on  the  Select  Questions  for  Evaluation  page. 

When  the  Conduct  Product  Evaluation  page  opens  (Figure  21),  the  user  will  be 
presented  with  a  list  of  questions  with  a  single  check  box  located  to  the  right  of  the 
question.  The  user  should  read  each  question  and  place  a  check  mark  next  to  each 
question  that  should  be  included  in  the  final  survey.  The  list  of  questions  on  this  form 
consists  of  the  questions  assigned  to  the  Product  Evaluation  section  that  meet  the  criteria 
established  in  the  Select  Questions  for  Evaluation  page. 
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When  all  questions  have  been  selected,  the  user  should  press  the  Generate  Survey 
button  to  create  the  survey  to  be  forwarded  to  the  personnel  in  the  field.  Selecting  the 
Close  Form  button  will  close  the  form  and  return  the  user  to  the  Product  Evaluation 
Options  Page  without  displaying  a  report. 
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Figure  2 1 .  Conduct  Product  Evaluation  Page 


When  the  user  presses  the  Generate  Survey  button  the  following  report  will  be 
generated  in  print  preview  mode.  Pressing  the  Printer  icon  will  send  the  report  to  the 
default  printer.  Pressing  the  Magnifying  Glass  icon  will  zoom  in  and  out  of  the  report. 
Pressing  the  Close  button  on  the  menu  bar  will  close  the  report  and  return  the  user  to  the 
Conduct  Alternatives  Analysis  page  (Figure  16). 
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Figure  22.  Product  Evaluation  Questionnaire 

10.  The  Exit  the  HSI  Evaluation  Tool  Button 

Pressing  the  Exit  the  HSI  Evaluation  Tool  button  on  the  Welcome  Screen  will 
close  ABRAHAM. 
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XI.  DISCUSSIONS,  CONCLUSIONS  AND  RECOMMENDATIONS 


The  Discussion,  Conclusions  and  Recommendations  chapter  begins  a  with 
discussion  of  conceptual  (the  question  base)  and  technical  (the  database)  challenges  the 
authors  faced  during  the  development  of  ABRAHAM.  Following  the  discussion  of  the 
HSI  Analysis  Tool  development  issues,  the  authors  present  the  feedback  obtained  during 
the  demonstration  sessions  at  the  HFE-TAG,  the  REF,  and  the  Marine  Corps  Operational 
Test  and  Evaluation  Activity  (MCOTEA).  Finally,  the  authors  summarize  the  work 
completed  to  date,  comment  on  the  degree  to  which  ABRAHAM  addresses  the  Effective 
Need  Statement,  and  conclude  with  recommendations  for  future  work  to  improve  the 
tool. 

A.  DISCUSSION 

During  the  development  of  ABRAHAM,  the  authors  identified  issues  related  to 
the  tool’s  concept  and  development.  This  section  explains  how  the  issues  influenced  the 
design  and  implementation  of  the  tool,  and  how  the  authors  decided  to  address  these 
concerns  in  the  current  iteration.  Unless  these  issues  are  appropriately  addressed  in 
future  iterations  of  ABRAHAM,  there  will  be  significant  threats  to  its  widespread 
acceptance  and  use. 

1.  Development  of  ABRAHAM 
a.  Conceptual  Issues 

During  the  development  of  ABRAHAM,  the  authors  identified  three 
conceptual  issues  that  resulted  in  significant  repercussions  for  the  project’s  success. 
Specifically,  the  authors  found  the  variety  of  system  types  and  lack  of  doctrine  made  it 
difficult  to  adopt  rules  for  question  implementation  and  scoring  systems.  Additionally, 
the  authors  struggled  to  define  a  meaningful  scoring  system  that  evaluated  the  HSI 
domains  and  presented  the  decision  maker  with  a  useful  recommendation.  Finally,  the 
general  language  used  to  apply  the  questions  to  multiple  categories  or  multiple  systems 
within  a  category  gives  rise  to  reliability  and  validity  concerns  for  the  questionnaires 
generated  by  the  tool.  A  more  thorough  discussion  of  these  conceptual  issues  follows. 
The  challenge  in  creating  a  tool  that,  by  definition,  is  so  general  in  nature  that  it  can  be 
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applied  to  multiple  systems  is  that  the  governing  rules  for  the  tool  must  also  be  fairly 
general.  Most  surveys  are  designed  to  evaluate  a  specific  piece  of  equipment  for  a 
specific  use  in  a  well  defined  environment,  for  example,  a  home  stereo  system  used  to 
project  surround  sound  in  a  home  entertainment  room.  In  this  scenario,  it  is  relatively 
easy  for  the  author  to  design  the  survey  based  on  the  equipment  and  environment  and 
analyze  the  responses  based  on  a  shared  understanding  of  the  environment  between 
author  and  respondent.  To  ensure  this  shared  understanding,  the  author  can  generate  very 
specific  questions  that  leave  little  room  for  misinterpretation. 

Using  ABRAHAM  to  create  surveys  that  leverage  shared  understanding 
between  REF  analyst  and  respondents  is  difficult  because  there  is  a  vast  number  of  the 
combinations  of  product  and  doctrine  that  could  be  evaluated.  For  example,  a  survey 
question  designed  to  ascertain  the  responsiveness  of  an  unmanned  aerial  vehicle  could 
mean  availability  to  Soldiers  on  reconnaissance  missions  in  Iraq  and  simple  flight 
worthiness  to  the  Soldiers  controlling  the  aircraft  in  high  altitude  environments  in 
Afghanistan.  The  authors  were  unable  to  identify  a  method  within  the  tool  to  establish  a 
rule  set  that  could  be  used  to  better  define  all  system  and  doctrine  combinations.  As  a 
result,  ABRAHAM  requires  the  REF  analyst  and  survey  respondents  maintain  strict 
adherence  to  the  REF  10-Liner  and  other  requirements  documents  as  the  underlying  basis 
for  evaluating  questions.  If  this  method  is  not  properly  managed  then  it  can  result  in 
inconsistent  responses  that  may  challenge  the  reliability  and  validity  of  the  survey 
instrument.  This  issue  is  the  primary  challenge  in  implementing  the  tool,  and  must  be 
reconsidered  in  any  future  implementation  of  this  database. 

The  second  conceptual  challenge  faced  was  developing  a  scoring  system 

that  provided  the  REF  analysts  and  decision  makers  with  a  “so  what”  statement  at  the  end 

of  the  analysis.  The  authors  made  allowances  for  the  REF  analysts  not  being  fully  trained 

HSI  specialists,  and  decided,  that  although  ABRAHAM  needed  to  provide  a  list  of  the 

HSI  issues  identified  for  the  system,  some  easy  to  understand  indication  of  the  “HSI 

worthiness”  was  also  needed.  The  authors  considered  a  points  system  and  a  traffic  light 

system  (red,  yellow,  green)  as  alternatives  for  the  indicator,  but  quickly  discovered  that 

there  was  no  standard  upon  which  to  base  this  recommendation.  In  a  commercial 

situation,  it  may  prove  useful  to  “red  light”  a  system  that  has  as  little  as  one  deficiency, 
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but  for  the  REF,  the  same  one  deficiency  may  mean  little  in  light  of  the  consequences  of 
not  fielding  the  equipment.  For  example,  even  if  the  REF’s  Packbot  exceeded  multiple 
military  or  OSHA  lift  standards,  it  may  be  far  better  to  field  the  equipment  with  its 
deficiencies  than  continue  to  let  Soldiers  prosecute  improvised  explosive  devices  by 
hand.  ABRAHAM  is  clearly  incapable  of  understanding  the  consequences  of  each 
situation,  and  the  use  of  a  traffic  light  or  other  scoring  methodology  within  the  tool  may 
prove  meaningless  if  most  of  the  fielded  REF  products  are  scored  in  the  “red  light”  or 
low  score  range.  As  discussed  in  the  Instruction  Manual  chapter,  in  the  current 
implementation,  a  percentage  of  the  maximum  total  score  is  the  method  used  to  convey 
the  measure  of  “HSI  worthiness”. 

The  third  major  conceptual  issue  faced  during  development  of  the  tool 
centered  on  the  reliability  and  validity  of  the  surveys  generated  by  the  tool.  As  discussed 
above,  there  are  many  challenges  to  creating  a  shared  understanding  of  system  and 
doctrine  between  the  REF  analyst  and  operators,  maintainers,  and  supporters  in  the  field. 
Without  the  shared  understanding  of  the  questions  and  what  the  survey  is  asking,  there 
should  be  little  expectation  that  the  survey  is  generating  the  information  it  was  designed 
to  extract.  Clearly,  the  validity  of  ABRAHAM  can  be  challenged,  and  if  the  survey  is 
without  validity  then  all  reliability  issues  are  moot. 

Unfortunately,  even  if  the  survey  instrument  generated  by  ABRAHAM  is 
valid,  the  Alternatives  Analysis  questionnaire  is  based  on  each  analyst’s  understanding  at 
the  time  of  survey  completion.  The  authors  have  little  expectation  for  inter-rater  and  test- 
retest  reliability  because  each  respondent’s  understanding  of  the  system  will  change  as 
they  leam  more  about  the  system,  doctrine,  environment  and  interactions  between  these 
variables.  REF  decision  makers  must  remember  that  the  Alternatives  Analysis  section  of 
HSI  Analysis  Tool  was  designed  to  identify  potential  HSI  risks  the  system  could  pose  to 
Soldiers  and  not  as  a  stand  alone  decision  support  aid. 

b.  Question  Development  Issues 

The  power  of  ABRAHAM  resides  in  its  ability  to  assist  REF  analysts 
(with  little  or  no  HSI  expertise)  to  conduct  comprehensive  HSI  evaluations  of  proposed 
systems.  While  the  development  of  the  HSI  question  base  should  have  been  the  easiest 
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part  of  the  overall  tool  development,  several  issues  related  to  question  development  arose 
during  the  project  that  limited  the  variety  of  questions  included  in  the  database.  A  brief 
discussion  follows  describing  these  issues,  including  consideration  for  the  types  of 
questions,  the  content  of  the  questions,  and  the  challenges  with  the  response  range. 

In  the  current  implementation,  ABRAHAM  allows  respondents  to  answer 
questions  using  closed-ended  answers  for  analysis  and  open-ended  answers  for  additional 
commentary.  As  discussed  in  the  Instruction  Manual  chapter,  the  methodology  for 
scoring  each  question  was  based  on  assigning  a  score  to  each  Likert  response  option  in 
order  to  later  identify  areas  of  unacceptable  risk.  This  scoring  method  allows  for  Likert 
scale  questions  with  ordered  responses,  where  each  response  can  be  graded  in  relation  to 
the  other  responses,  but  not  categorical  responses.  For  example,  ABRAHAM  is  designed 
to  analyze  a  question  in  which  the  response  measures  are  “up  to  20%”,  “21%  to  40%”, 
“41%  to  60%”,  “61%  to  80%”,  and  “81%  to  100%”,  but  the  tool  is  not  designed  to 
analyze  a  question  in  which  the  response  measures  do  not  imply  value,  such  as  the 
demographic  responses  of  “African  American”,  “Asian/Pacific  Islander”,  “Caucasian”, 
“Hispanic”,  or  “Other”.  ABRAHAM  is  capable  of  analyzing  binary  questions  as  a  subset 
of  the  five  point  Likert  scale,  but  it  is  not  capable  of  understanding  ordinal  responses  in 
which  respondents  are  asked  to  rank  a  set  of  options.  ABRAHAM  is  also  incapable  of 
analyzing  Constant  Sum  responses,  Stapel  Scale  responses,  Semantic  Differential  Scale 
responses,  and  Multiple  Choice  responses.  Open-ended  responses  are  provided  to  allow 
respondents  to  convey  additional  information,  but  these  responses  are  not  automatically 
scored  by  ABRAHAM. 

As  the  authors  began  developing  the  question  base,  it  quickly  became 
apparent  that  two  categories  of  questions  were  needed  to  fully  address  Alternatives 
Analysis  and  Product  Evaluation.  Although  there  were  some  questions  that  applied  to 
both  questionnaires,  Alternatives  Analysis  questions  tended  to  compare  equipment 
specifications  to  HSI  domain  standards  while  Product  Evaluation  Questions  tended  to 
query  the  usability,  reliability,  maintainability,  and  other  “ilities” 
(http://www.answers.com/topic/ilities)  related  to  the  equipment  and  its  training  systems. 
Alternatives  Analysis  questions  were  generated  from  the  knowledge  gathered  from  the 
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HSI  domain  literature  reviews  and  input  from  SMEs.  Product  Evaluation  questions  were 
derived  from  SME  input,  general  experience,  and  previous  Anny  surveys  and 
questionnaires. 

The  final  issue  with  question  development  concerns  the  possible  range  of 
responses  that  may  be  required  in  the  Product  Evaluation  questions.  For  example,  if  the 
database  contains  a  question  that  assesses  the  accuracy  of  a  weapon  system,  a  response 
scale  measured  in  millimeters  may  be  appropriate  for  a  rifle  but  completely  inadequate 
for  long-range  artillery.  The  current  implementation  of  ABRAHAM  requires  a  separate 
question  if  the  response  requires  a  different  scale.  This  issue  raises  a  concern  because  it 
can  slow  the  analysis  process  and  dissuade  the  REF  analysts  from  using  the  tool. 

c.  Database  Development  Issues 

There  were  three  programmatic  challenges  that  limited  overall 
functionality  of  the  first  implementation  of  ABRAHAM’s  database  and  may  prove 
difficult  to  address  in  future  iterations  of  program.  The  most  critical  database  issue  the 
authors  faced  was  the  inability  to  dynamically  build  new  tables  to  store  responses  to  the 
Product  Evaluation  Questionnaire.  Secondly,  the  authors  could  not  develop  an 
acceptable  methodology  for  limiting  the  total  number  of  questions  in  the  Alternatives 
Analysis  and,  therefore  to  limit  the  total  time  required  to  complete  the  survey  while  also 
meeting  the  goal  of  providing  a  comprehensive  analysis  of  all  HSI  domains.  Finally,  the 
authors  were  unable  to  develop  a  method  of  storing  the  queries  that  were  used  to  generate 
the  surveys  for  future  use  by  REF  analysts.  These  issues  will  now  be  covered  in  greater 
depth. 

When  developing  a  new  table  to  store  data  in  Microsoft  Access,  the 
program  requires  that  the  programmer  define  the  dimensions  of  the  table.  In  Access,  the 
programmer  must  specify  the  number  of  columns  (fields)  and  define  the  type  of 
information  each  column  will  contain.  The  number  of  rows  (records)  can  number  in  the 
thousands  and  does  not  have  to  be  specified  when  the  table  is  first  designed.  The 
challenge  with  ABRAHAM  is  that  the  number  of  questions  in  each  survey  is  variable. 
Additionally,  for  each  piece  of  equipment  to  be  evaluated,  the  numbers  of  respondents  to 
each  survey  can  be  variable.  Ideally,  the  REF  analyst  should  be  able  to  generate  a  new 
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table  to  store  survey  responses  at  the  click  of  a  button,  but  there  is  no  provision  for  this  in 
the  current  iteration  of  ABRAHAM.  Without  the  ability  to  dynamically  create  this  new 
table,  ABRAHAM  is  unable  to  store  Product  Evaluation  responses  and  cannot  be  used  to 
compute  the  summary  and  descriptive  statistics  that  would  provide  insight  into  the 
product’s  performance  in  the  field.  Although  the  ideal  solution  would  be  to  develop  a 
“push  button”  method  previously  discussed  for  a  variable  number  of  fields,  one  other 
possible  solution  would  be  to  limit  all  surveys  to  a  specific  number  of  questions  or  a 
specific  number  of  respondents  to  allow  for  pre-programming  of  ABRAHAM  to  create 
the  appropriate  tables  to  save  the  data  at  the  push  of  a  button. 

As  the  question  base  of  ABRAHAM  grows,  the  program  must  strike  a 
balance  between  limiting  the  number  of  questions  asked  in  order  to  meet  REF  time 
constraints  and  asking  enough  questions  to  provide  a  sufficient  look  at  the  HSI  decision 
space.  In  the  Product  Evaluation  section  of  ABRAHAM,  the  REF  analyst  serves  as  a 
filter  to  limit  the  final  questionnaire  to  select  only  those  questions  that  are  most 
applicable  to  the  Soldiers  in  the  field.  In  the  Alternatives  Analysis  section  of 
ABRAHAM,  the  REF  analyst  is  presented  with  all  questions  (based  on  the  choices 
selected  on  the  Questions  for  Evaluation  page)  because  there  is  currently  no  computer- 
based  filter  to  limit  the  number  of  questions  asked.  While  there  is  no  technological 
limitation  that  would  require  a  limitation  on  the  number  of  questions  asked,  it  is  generally 
infeasible  for  the  REF  analyst  to  spend  more  than  a  few  hours  on  these  surveys.  During 
database  development,  the  authors  discussed  methods  for  limiting  the  number  of 
questions,  including  a  simple  random  sampling  of  the  question  base,  a  stratified  random 
sampling  of  the  question  base  based  on  question  importance  and  HSI  domain,  and  a 
method  of  selecting  all  high  importance  questions  and  a  percentage  of  lower  importance 
questions  to  arrive  at  a  predetermined  number  of  questions.  The  current  version  of 
ABRAHAM  does  not  implement  any  of  these  filtering  schemes  because  the  authors  felt  it 
was  important  to  identify  HSI  concerns  using  an  unfiltered  system,  especially  with  the 
question  base  in  its  infancy. 

The  final  major  challenge  faced  in  development  of  ABRAHAM  was 

establishing  the  ability  to  save  the  database  queries  used  to  generate  the  Alternatives 

Analysis  and  Product  Evaluation  surveys.  ABRAHAM  currently  uses  the  Microsoft 
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Query  by  Form  methodology  to  provide  an  intuitive  interface  for  tool  users,  but  there  is 
no  method  in  place  to  save  this  input.  Without  the  ability  to  save  the  database  queries,  the 
REF  analyst  cannot  recall  a  survey  to  reevaluate  a  piece  of  equipment,  especially  if  a 
random  sampling  filter  is  employed  to  limit  the  number  of  questions.  This  is  especially 
troublesome  if  the  REF  analyst  wants  to  use  a  previously  created  survey  to  evaluate  a 
second  or  subsequent  iteration  of  a  piece  of  equipment  to  detennine  if  changes  to  the 
system  were  beneficial. 

Once  software  development  reached  a  level  that  allowed  for  the  basic  operation  of 
the  module,  the  authors  arranged  demonstrations  with  the  REF  and  other  HSI  domain 
experts  to  gamer  feedback  and  determine  the  initial  response  to  the  tool.  The  following 
section  discusses  the  authors’  reasons  for  selecting  the  demonstration  venues,  a 
description  of  the  demonstration  at  each  venue,  and  a  discussion  of  the  feedback  obtained 
at  each  presentation. 

2.  HSI  Analysis  Tool  Demonstration 

The  ABRAHAM  prototype  was  demonstrated  in  three  separate  venues  within  a 
two  week  time  period  from  6-20  November  2006.  The  authors  selected  the  DoD 
Human  Factors  Engineering  Technical  Advisory  Group  (HFE  TAG),  the  REF,  and  the 
USMC  Operational  Test  and  Evaluation  Activity  because  these  sites  provided  access  to 
HSI  practitioners  who  had  a  broad  range  of  expertise  within  each  of  the  domains  of  HSI 
and/or  who  understood  the  needs  of  customer  activities.  Since  HSI  is  a  relatively  new 
field  and  one  that  represents  an  integration  of  various  established  fields,  HSI-focused 
projects  must  incorporate  each  domain.  Without  fair  representation  of  each  domain  in  the 
database,  optimization  is  less  likely.  A  second  reason  for  selecting  three  demonstration 
venues  was  that  acceptance  from  more  than  one  venue  validated  the  proof  of  concept  and 
helped  to  ensure  the  database’s  utility.  Demonstrating  the  tool  to  just  the  REF  would 
possibly  have  resulted  in  a  tool  that  was  too  specific  to  the  REF  and  its  current  processes. 
By  gathering  diverse  input,  the  authors  were  able  to  shape  ABRAHAM  to  meet  the 
REF’s  need  while  remaining  generalized  enough  to  support  other  organizations  that 
expressed  an  interest  in  the  methodology.  The  authors  felt  this  characteristic  of 
ABRAHAM  was  essential,  given  recent  efforts  and  guidance  for  the  services  within  DoD 
to  adapt  a  joint  prospective. 
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a.  The  HFE  TA  G  Demonstration  Session 

The  first  presentation  of  the  database  was  at  the  HFE  TAG  conference 
located  in  Monterey,  California  on  6  November  2006.  This  presentation  was  an  infonnal 
poster  session  and  demonstration  of  ABRAHAM’s  concept  and  contents.  The  audience 
included  Human  Factors  Engineering  and  HSI  experts.  During  this  time,  experts  were 
able  to  provide  the  authors  with  additional  information  as  well  as  view  a  demonstration  of 
ABRAHAM.  Throughout  the  two  hour  demonstration  period,  small  groups  were 
provided  with  a  walkthrough  of  the  tool.  In  sum,  the  demonstration  was  given 
approximately  12  times  to  audiences  that  represented  the  U.S.  Army,  U.S.  Air  Force, 
U.S.  Navy,  U.S.  Marine  Corps,  Canadian  military  services,  and  civilian  contractors. 

Overall,  feedback  from  domain  experts  was  extremely  positive. 
Specifically,  experts’  comments  demonstrated  support  for  the  database’s  concept,  design, 
content,  and  abilities.  Domain  experts  were  pleased  with  the  customized  literature  review 
that  was  tailored  to  rapid  acquisition  and  the  REF.  Concerns  were  limited  to  the 
database’s  possible  growth  and  transition  into  the  traditional  acquisition  field. 
Specifically,  the  attempt  to  generalize  questions  to  capture  the  broad  domains  of  HSI  has 
not  always  rendered  usable  results  or  tools. 

b.  Presentation  to  the  Rapid  Equipping  Force 

As  mentioned  previously,  the  primary  customer  of  ABRAHAM  was  the 
Army’s  Rapid  Equipping  Force.  The  authors  traveled  to  Fort  Belvoir,  Virginia  to  present 
the  current  state  of  the  research  to  the  REF  20  November  2006.  The  purpose  of  this 
meeting  was  threefold.  First,  the  authors  wanted  to  ensure  that  the  database’s  concept 
and  contents  met  the  customer’s  needs  and  desires.  Second,  the  authors  wanted  to  obtain 
customer’s  feedback  for  necessary  modifications  and  product  improvement.  Finally,  at 
the  conclusion  of  the  meeting,  the  authors  sought  agreement  with  the  REF  on  the  next 
steps  that  should  be  taken. 

The  REF  meting  began  with  introduction  of  all  attendees,  which  included:: 

•  John  Geddes,  Director,  Technology  Management  Director 

•  LTC  William  Garland,  General  Support  Product  Manager 

•  Joe  Rozmeski,  Deputy,  Operations 
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•  Scott  Torgerson,  Capabilities  &  Assessments  Team  Lead 

•  Betty  Maguder,  Assessment  Analyst 

•  MAJ  Bob  Lentz,  Assessment  Analyst 

The  presentation  began  with  a  discussion  of  the  researchers’  backgrounds, 
a  description  of  HSI,  and  a  discussion  of  how  HSI  should  apply  to  the  REF’s  overall 
analysis  methodology.  Once  the  presentation  was  completed,  the  ABRAHAM 
demonstration  began  and  open  discussion  was  welcomed.  The  following  four  paragraphs 
highlight  the  REF’s  primary  concerns  and  feedback. 

The  REF  expressed  great  satisfaction  with  ABRAHAM’s  intuitive  user 
interface  and  easy-to-understand  navigation  aids.  The  REF  expressed  a  great  deal  of 
satisfaction  with  the  ease  with  which  the  Product  Evaluation  section  was  able  to  generate 
tailored  surveys  and  liked  the  ability  to  exclude  any  question  on  the  survey  that  was 
inappropriate  for  the  system.  Although  the  REF  analysts  were  equally  pleased  with  the 
Alternatives  Analysis  section  of  the  tool,  questions  were  raised  about  the  sensitivity  and 
utility  of  the  scoring  system.  Suggested  areas  of  improvement  included:  the  refinement  of 
the  product  categories  to  more  closely  match  REF  projects;  a  method  to  generate  separate 
Product  Evaluation  surveys  for  operators,  maintainers,  and  supervisors;  and  an  ability  to 
distribute  surveys  via  personal  digital  assistant  (PDA)  or  via  the  Internet. 

The  REF  recommended  two  major  improvements  to  ABRAHAM  that  if 
incorporated,  would  greatly  improve  the  analysis  capabilities  of  the  command.  The  first 
recommendation  involved  having  the  HSI  tool  automatically  compute  summary  and 
descriptive  statistics  for  Product  Evaluation  that  could  be  used  in  briefs  and  reports 
provided  to  decision  makers.  The  second  recommendation  involves  implementing 
ABRAHAM  in  a  networked  environment  to  allow  analysts  to  share  a  common  question 
base  and  leverage  the  combined  knowledge  of  all  analysts  to  grow  and  mature  the  tool. 

As  previously  discussed,  employing  a  statistical  analysis  algorithm  in 
ABRAHAM  is  technically  feasible,  but  implementing  this  feature  under  the  current 
iteration  of  the  tool  would  require  a  major  break  in  the  user  interface  to  define  the 
dimensions  of  the  table  that  would  store  the  data.  The  authors  recommended  that  REF 


101 


analysts  employ  a  commercially  available  statistical  analysis  package  such  as  SPSS,  S- 
Plus,  or  SAS  until  the  feature  can  be  implemented  in  ABRAHAM. 


The  current  iteration  of  ABRAHAM  was  not  designed  to  support  user 
accounts  and  other  network  specific  applications,  but  the  tool  can  be  employed  in  a  multi¬ 
user  environment  as  coded  with  the  caveat  that  no  two  users  could  have  the  same 
question  record  (row  of  a  table)  open  at  a  time.  Microsoft  Access  is  capable  of  limited 
employment  in  a  networked,  multi-user  environment,  but  the  program  can  become 
unreliable  as  the  number  of  users  increase.  A  more  complete  discussion  of  multi-user 
database  environments  is  included  in  the  Conclusion  section  of  this  chapter. 

c.  Presentation  to  MCOTEA 

The  final  presentation  was  conducted  at  MCOTEA  at  Quantico,  Virginia, 
on  20  November  2006.  This  presentation  was  conducted  in  response  to  the  Marine 
Corp’s  request  for  a  more  robust  HSI  questionnaire  database  that  would  accommodate 
their  test  and  evaluation  methodology.  Similar  to  the  presentation  given  to  the  REF,  the 
presentation  commenced  with  a  discussion  about  the  researchers’  backgrounds  and  a 
description  of  HSI,  which  included  the  importance  of  HSI  and  how  HSI  fits  into  the 
MCOTEA’s  overall  methodology  of  analyses. 

Feedback  from  MCOTEA  focused  on  the  differences  in  language  and 
tenninology  between  traditional  and  rapid  acquisitions  environments  within  the  Army 
and  Marine  Corps.  Due  to  the  differences  in  word  usage  between  the  services,  the  ability 
of  ABRAHAM  to  immediately  generalize  from  the  Anny  to  Marine  Corp  setting  is 
somewhat  limited.  This  shortfall  can  be  addressed  through  a  more  tailored  database  for 
each  service. 

MCOTEA  also  noted  the  absence  of  statistical  analysis  and  questioned  the 
sensitivity  of  and  requirement  for  the  Alternatives  Analysis  scoring  mechanism.  Again, 
the  authors  recommended  that  MCOTEA  analysts  employ  a  commercially  available 
statistical  analysis  package  such  as  SPSS,  S-Plus,  or  SAS  until  the  feature  can  be 
implemented  in  ABRAHAM. 
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B. 


CONCLUSION 


In  summary,  ABRAHAM  was  developed  to  address  the  needs  of  the  Rapid 
Equipping  Force’s  need  for  a  Human  System  Integration  evaluation  method.  The 
creation  of  this  Microsoft  Access  database  application  was  initiated  to  assist  in  quickly 
assessing  and  fielding  new  products  for  Combatant  Commanders  conducting  operations 
in  the  rapidly  changing  and  increasingly  complex  Global  War  on  Terrorism. 

ABRAHAM  is  capable  of  generating  tailored  surveys  to  evaluate  REF  equipment 
by  prompting  the  analyst  to  classify  the  new  system  and  by  applying  it  to  equipment 
categories  within  the  tool.  Based  on  this  classification,  ABRAHAM  selects  applicable 
questions  from  its  question  base  and  generates  a  survey  for  either  Alternatives  Analysis 
or  Product  Evaluation.  In  the  current  iteration  of  ABRAHAM,  Alternatives  Analysis 
survey  responses  are  evaluated  to  identify  HSI  areas  of  high  risk  to  equipment  operators, 
maintainers  and  supporters. 

ABRAHAM  was  showcased  in  three  Department  of  Defense  settings,  the 
Department  of  Defense  Human  Factors  Engineering  Technical  Advisory  Group,  the 
Army’s  Rapid  Equipping  Force,  and  the  United  States  Marine  Corps’  Operational  Test 
and  Evaluation  Activity.  During  each  demonstration,  a  complete  walkthrough  of 
ABRAHAM  was  conducted  and  feedback  from  stakeholders  and  subject  matter  experts 
was  requested.  In  each  instance,  the  authors  received  positive  feedback  and  multiple 
requests  for  immediate  implementation. 

The  overall  concept  and  content  of  ABRAHAM  appears  to  fill  a  gap  in  the  current 
library  of  HSI  tools.  Based  on  the  feedback  provided  during  the  product  demonstrations, 
it  has  been  determined  that  there  is  sufficient  interest  to  further  develop  this  tool  both  the 
traditional  and  rapid  acquisition  processes.  Although  the  limitations  of  the  current  form 
of  ABRAHAM  prevent  the  implementation  of  critical  analysis  features,  future  iterations 
can  easily  build  upon  the  foundation  and  effectively  address  the  concerns  identified  at 
each  product  demonstration. 

The  next  iteration  of  ABRAHAM  should  focus  on  refining  the  current 
implementation  of  the  tool  in  Microsoft  Access.  Specifically,  the  tool  should  be  modified 
to  allow  for  the  Product  Evaluation  Survey  to  be  broken  out  by  surveys  specifically 
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designed  for  operators,  maintainers  and  supervisors.  Additionally,  equipment  categories 
within  the  tool  should  be  modified  to  meet  the  equipment  fielded  by  the  REF  and 
MCOTEA.  Finally,  testing  of  ABRAHAM  in  a  networked  environment  should  be 
conducted  to  determine  the  stability  of  the  tool  in  a  limited  multi-user  environment. 
These  improvements  to  ABRAHAM  can  be  implemented  immediately  in  Microsoft 
Access  with  little  or  no  development  costs. 

If  funding  is  provided  for  further  development  of  ABRAHAM  then  the  following 
improvements  should  be  considered.  First,  Microsoft  Access  provides  a  limited  multi¬ 
user  environment,  but  the  program  can  become  unstable  and  unreliable  as  the  database 
file  size  and  number  of  users  increase.  Although  Microsoft  advertises  that  Access  can 
support  up  to  255  users,  it  is  best  suited  for  small  single  user  desktop  applications.  At 
best,  Microsoft  Access  is  capable  of  reliably  accommodating  approximately  10  users  with 
diminishing  stability  as  the  database’s  file  size  increases.  Therefore,  it  is  recommended 
that  ABRAHAM  be  redesigned  for  use  in  a  true  multi-user  database  environment  such  as 
Oracle,  Microsoft  SQL,  MySQL,  FoxPro,  or  Sybase. 

Use  of  one  of  these  programs  will  likely  require  professional  programmers  who 
should  be  capable  of  addressing  the  challenges  that  limited  the  current  Access 
implementation  of  ABRAHAM.  Most  importantly,  the  programmers  must  be  able  to 
identify  a  method  to  store  survey  results  in  order  to  allow  for  automatic  statistical 
analysis.  Another  important  issue  that  programmers  will  need  to  address  in  a  multi-user 
environment  is  providing  an  ability  to  identify  and  consolidate  duplicate  question  entries 
to  keep  ABRAHAM  streamlined  and  manageable. 

In  addition  to  the  programming  changes  to  ABRAHAM,  further  research  should 
be  conducted  to  detennine  the  most  effective  method  for  communicating  “HSI- 
worthiness”  in  the  Alternatives  Analysis  report.  Future  work  on  the  tool  should  also 
focus  on  determining  if  limiting  the  number  of  questions  is  warranted  and  if  so  then 
establishing  a  methodology  for  selecting  the  best  questions  for  each  survey.  Clearly,  this 
work  should  be  completed  before  investing  in  professional  programmers  to  convert  the 
tool. 
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In  order  to  grow  the  database  of  questions  within  ABRAHAM,  a  class  project 
should  be  implemented  in  a  course  such  as,  OA3402,  Research  Methods  for  Perfonnance 
Assessment,  that  requires  each  student  to  develop  five  or  ten  Likert  scale  survey 
questions  that  are  based  on  established  guidelines  within  each  of  the  domains  of  HSI  as 
well  as  in  the  areas  of  reliability,  usability,  and  lethality.  This  class  project  can  be 
assigned  as  homework  that  supports  chapters  addressing  survey  methods.  The 
assignment  would  not  interfere  with  classroom  instruction.  The  strength  of  ABRAHAM 
is  based  on  the  quality  of  questions  in  the  database,  and  leveraging  the  knowledge  and 
experience  of  the  Naval  Postgraduate  School  student  body  can  only  improve  the  product. 

There  are  two  areas  for  future  thesis  work  on  ABRAHAM.  The  first  thesis 
opportunity  involves  evolving  ABRAHAM  from  an  application  that  is  primarily  geared 
towards  the  Army’s  Rapid  Equipping  Force  to  a  tool  that  addresses  both  rapid  and 
traditional  acquisitions  in  the  joint  service  environment.  The  second  thesis  opportunity 
involves  developing  recommendations  for  further  analysis  for  each  question  in  the 
database. 
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APPENDIX  A.  DATABASE  QUESTIONS 


DOCTRINE 

Does  the  use  of  the  item  improve  the  speed  of  the  current  battle  drill? 

much  worse  the  same  slightly  much  better 

better 

How  many  missions  have  you  taken  the  item  on? 

In  what  way  did  you  employ  the  item  during  training  or  at  home  station? 

How  do  you  plan  on  using  the  item  when  you  deploy? 


Did  the  employment  of  this  system  alter  your  standard  TTPs? 


Did  the  employment  of  the  item  alter  your  standard  TTPs  for  room  entering/ 
clearing  (or  anything  else)? 


How  useful  is  the  Concept  of  Operations  (CONOPS)  and  Tactics,  Techniques,  and 
Procedures  (TTP)  information  provided  with  the  system? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

How  many  times  (missions)  have  you  used  this  system? 


Does  the  system  require  new  doctrine? 

Yes  No 

The  system  significantly  enhanced  your  mission  success. 

strongly  disagree  neutral  agree  strongly 

disagree  agree 
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Did  this  system  meet  your  unit  requirements? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Other  than  for  the  system  description,  describe  any  other  uses  you  found  for 
using  any  of  the  components  of  this  system: 


If  no  relevant  procedures  are  in  place,  to  what  extent  would  developed  and 
implemented  procedures  provide  either  a  complete  or  partial  solution  to  the  issue 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Is  there  a  valid  need  for  your  organization  to  have/use  this  system? 

No  Yes 

Explain  generally  where  (AOR),  when  (dates  used),  by  who  (unit  type(s)),  and 
how  (CONOPS  or  TTPs)  the  system  was  used. 


In  what  type  of  area  was  the  system  employed?  (Open  Area,  FOB,  Urban) 


To  what  extent  did  the  system  meet  the  unit’s  requirement? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Describe  the  extent  to  which  the  unit  needs  this  system. 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Environment,  Safety  and  Occupational  Health 

What  extreme  weather,  to  include  winds/sand  storms,  was  system  operated  in? 


What  were  the  typical  weather  conditions? 


How  many  times  have  you  either  had  to  activate  this  system  yourself  or  it 
activated  itself,  because  of  a  vehicle  fire 


Did  you  have  any  issues  with  riding  in  the  vehicle  without  damaging  the  system? 

prevents  big  issues,  noticeable  small  issues  no  issues 
from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 

At  what  type  of  location  did  you  employ  the  system? 


Do  you  have 

any  issues  with  the  your  ability  to  secure  yourself  while  wearing  the 

system? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Do  you  have 

any  issues  with  ingress/egress  through  ramp/ramp  doors  (trips, 

snags,  catches) 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

To  what  extent  does  the 

system  shield  the 

user  from  radiation  when  transmitting? 

To  no 

To  a  small 

To  a 

To  a  great 

To  a  very 

extent 

extent 

moderate 

extent 

extent 

great  extent 

What  was  the  approximate  average  temperature  (in  degrees  Fahrenheit) 


FACILITIES 

To  what  extent  is  the  leadership  receptive  to  the  user  of  the  new  system? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderated  extent  great  extent 

extent 
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Specify  the  forward  operating  base  where  the  barrier  system  is  used? 


To  what  extent  does  inadequate  infrastructure  impact  the  issue? 

To  a  very  To  a  great  To  a  To  a  small  To  no 

great  extent  extent  moderate  extent 

extent 

Will  the  system  require  added  special  facilities  for  storage? 

Yes  No 

HABITABILITY 

Rate  the  ease  of  putting  on  and  taking  off  the  shirt: 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Rate  the  comfort  of  the  shirt: 

prevents  big  issues,  noticeable 

from  doing  hard  to  do  issue, 

your  job  my  job  affects  my 

job 

What  is  the  time  of  day  (hours)  when  you  typically  wear  this  shirt 


small  issues  no  issues 
but  easy  to 
live  with 


How  many  hours  do  you  typically  wear  this  shirt  when  you  wear  it? 


How  many  times  have  you  worn  this  shirt? 


Rate  the  comfort 


prevents 

big  issues, 

noticeable 

small  issues  no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 
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Rate  the  laundering  ability  of  the  shirt 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Do  you  feel  comfortable  and  confident  using  this  product? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Does  the  system  prevent  you  from  fitting  inside  the  vehicle  when  wearing? 

prevents  big  issues,  noticeable  small  issues  no  issues 

from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 

Did  you  have  any  issues  with  comfort  while  riding  in  a  vehicle  with  the  with  the 


system  donned? 

prevents 

big  issues, 

noticeable 

small  issues  no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

HUMAN  FACTORS  ENGINEERING 

Rate  the  effectiveness  of  the  weapon  adaptor  and  suppressor  mounting  mechanism 


Very  Ineffective 

Neither  Effective 

Very 

ineffective 

ineffective 

effective 

nor  effective 

What  components  need  to  be  improved  and  how? 


Rate  the  durability  of  the  mounting  system 


Very 

ineffective 

Ineffective 

Neither 
ineffective 
nor  effective 

Effective 

Very 

effective 

Rate  the  ruggedness  of  the 

cables 

Very 

ineffective 

Ineffective 

Neither 
ineffective 
nor  effective 

Effective 

Very 

effective 
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Can  the  font  size  be  adjust  in  a  reasonable  amount  of  time? 

No  Yes 

Can  the  font  size  be  adjusted? 

No  Yes 

To  what  extent  is  the  font  size  of  the  system  legible  during  operations 
conditions? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 


Rate  the  reliability  of  running  this  item  off  of  the  vehicle  battery 


completely 

unreliable, 

noticeable 

small  issues  no  issues 

unreliable, 

hard  to  do 

issue, 

but  easy  to 

prevents 

my  job 

affects  my 

live  with 

from  doing 

job 

your jo 

What  components  should  be  substituted  for  using  other  similar  types  of 
equipment  (include  specific  suggestions  for  substitution)? 


To  what  extent  does  the  system  require  added  coordination  between  systems? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Rate  the  ease  of  installing  the  system 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Approximately  how  many  rounds  did  you  shoot  through  the  suppressor? 


Did  this  system  ever  become  inoperable  or  ineffective? 

Always  Most  of  the  Half  of  the  Occasionall  Never 

time  time 
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Did  this  system  interfere  with  the  operation  of  the  weapon  to  which  it  was 
mounted? 


prevents 
from  doing 
your job 

big  issues, 
hard  to  do 
my  job 

noticeable 
issue, 
affects  my 
job 

small  issues 
but  easy  to 
live  with 

no  issues 

Is  the  system  checklist  easy  to  understand? 

Very 

difficult 

Difficult 

Neither 
difficult  nor 

Easy 

Very  easy 

easy 

Does  the  system  clearly  identify  errors? 


No  Yes 

Are  there  multiple  methods  of  input  for  users? 

No  Yes 

Were  the  cables  easy  to  manage  and  maintain? 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Were  there  any  problems  with  the  laptop? 

prevents  big  issues,  noticeable  small  issues  no  issues 

from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 

Was  battery  charger  (system/laptop)  an  issue  with  extreme  hot/cold  conditions? 

Yes  No 

Is  the  system  design  suitable  for  the  battle  field? 

No  Yes 

On  average,  how  long  did  the  filters  last  between  cleanings? 
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Rate  the  ease  of  changing  the  batteries 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Rate  the  ease  of  putting  the  item  on 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

On  average,  how  long  did  the  batteries  last? 


Can  the  backlight  of  the  system  be  adjusted? 

No  Yes 

Were  there  any  interoperability  issues? 

prevents  big  issues,  noticeable  small  issues  no  issues 

from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 

Rate  the  maintainability  of  the  system  (easily  cleaned,  fixed,  etc.) 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Rate  the  reliability  of  the  system  (daily  wear  and  tear,  did  it  work  consistently?): 

Very  Ineffective  Neither  Effective  Very 

ineffective  effective  effective 

nor 

Rate  the  ease  of  uninstalling  the  equipment 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

What  components  need  to  be  eliminated  and  why? 
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Rate  the  ease  of  operating  the  system 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 

Rate  the  ease  assembling 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 

If  this  system  is  fielded,  would  you  use  it/wear  it? 


To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Describe  your  experience  with  the  this  system  (how  many  times  have  you  used  it, 
during  what  types  of  missions,  at  what  time  of  day/night)? 

Did  this  system  meet  your  mission  needs? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Did  this  system  significantly  enhance  your  mission  success? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 


Rate  the  ease  of  using  this  system? 


Very 

Difficult 

Neither 

Easy 

Very  easy 

difficult 

difficult  nor 

easy 

Does  the  system  increase  your  Situational  Awareness? 

To  no 

To  a  small 

To  a 

To  a  great 

To  a  very 

extent 

extent 

moderate 

extent 

extent 

great  extent 
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At  what  range  did  it  perform  to,  in  meters? 


Are  you  able  to  determine  that  all  circuits,  adjacent  circuits,  charged  capacitors, 
or  equipment  to  be  repaired  are,  in  fact,  de-energized? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

What  components  need  to  be  added? 


How  many  times  did  you  clean  the  filters? 


Rate  the  ease/difficulty  of  the  systems  ability  to  create  and  send  messages 

impossible  fairly  neutral  fairly  easy  no  problems 

difficult  at  all 

To  what  extent  was  the  antenna  effective  in  receiving  signals? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Is  the  software  provided  adequate  for  the  mission? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Rate  the  clarity  of  the  text,  symbols  and  icons  of  the  software? 


completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 

adequate 

adequate 

Rate  the  size 

of  the  text? 

completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 

adequate 

adequate 

Rate  the  size 

of  the  symbols? 

completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 

adequate 

adequate 
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Were  the  symbols  distinguishable  and  easy  to  interpret? 


completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 

adequate 

adequate 

Was  the  lay  of  the  screen 

completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 

adequate 

adequate 

Do  you  have  any  problems 

with  using  the 

:  buttons  by 

feel  and  not  sight? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Navigating  throughout  the 

program  was 

completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 

adequate 

adequate 

Do  you  have 

any  issues  with  mounting/dismounting  subsystems  of  the  system? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Rate  the  ease/difficulty  of  the  systems  ability  to  use  map  zooming  and  panning 

impossible 

fairly 

neutral 

fairly  easy 

no  problems 

difficult 

at  all 

Rate  the  ease/difficulty  of  the  systems  ability  to  switch  between  maps 

impossible 

fairly 

neutral 

fairly  easy 

no  problems 

difficult 

at  all 

Rate  the  ease/difficulty  of  the  systems  ability  to  backtrack  and  fix  identified 

errors 

impossible 

fairly 

neutral 

fairly  easy 

no  problems 

difficult 

at  all 

Rate  the  ease/difficulty  of  monitoring  the  systems  status  (battery  life,  signals 

strength) 

impossible 

fairly 

neutral 

fairly  easy 

no  problems 

difficult 

at  all 
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Rate  the  ease/difficulty  of  the  system's  ability  to  be  configured  to  operate  with 
other  systems. 


impossible 

fairly 

difficult 

neutral 

fairly  easy 

no  problems 
at  all 

Rate  the  ease/difficulty  of  performing  maintenance  on  the  system 

impossible 

fairly 

difficult 

neutral 

fairly  easy 

no  problems 
at  all 

Rate  the  ease/difficulty  of  troubleshooting  the  system 

impossible 

fairly 

difficult 

neutral 

fairly  easy 

no  problems 
at  all 

The  menu  item  across  the  were 

completely 

inadequate 

somewhat 

inadequate 

neutral 

somewhat 

adequate 

completely 

adequate 

Does  the  system  prevent 

your  range  of  motion? 

prevents 
from  doing 
your job 

big  issues, 
hard  to  do 
my  job 

noticeable 
issue, 
affects  my 
job 

small  issues 
but  easy  to 
live  with 

no  issues 

Do  you  have 

any  issues  with  the  ability 

to  check  and  change  your  batteries? 

prevents 
from  doing 
your job 

big  issues, 
hard  to  do 
my  job 

noticeable 
issue, 
affects  my 
job 

small  issues 
but  easy  to 
live  with 

no  issues 

Do  you  have 

any  issues  with  the  weight 

of  this  system? 

prevents 
from  doing 
your job 

big  issues, 
hard  to  do 
my  job 

noticeable 
issue, 
affects  my 
job 

small  issues 
but  easy  to 
live  with 

no  issues 

Does  this  system  change 

the  way  you  wear  your  gear? 

prevents 
from  doing 
your job 

big  issues, 
hard  to  do 
my  job 

noticeable 
issue, 
affects  my 
job 

small  issues 
but  easy  to 
live  with 

no  issues 
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Do  you  have  any  issues  with  your  ability  to  wear/  use  this  system  with  existing 


gear? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Do  you  have 

any  issues  the  headset  comfort  after  extended  wear? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Do  you  have 

any  issues  with  compatibility  of  the  helmet? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Do  you  have 

any  issues  with  understanding  radio  comms? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Does  the  system  prevent 

you  from  mounting  and  dismounting  the  exterior  of  a 

vehicle? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Do  you  have  any  issues  manipulating  the  system's  buttons  with/without  gloves 

and  in  all  weather? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

The  software  feedback  helps  to  prevent 

or  identify  operator  errors 

completely 

somewhat 

neutral 

somewhat 

completely 

disagree 

disagree 

agree 

agree 

119 


Does  the  system  prevent  you  from  assuming  the  correct  firing  position? 


prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Does  the  system  prevent 

you  from  firing 

around  corner/out  of  windows? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Does  the  system  prevent 

you  from  performing  individual  movements  (crawls, 

climbs)? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Does  the  system  prevent 

you  from  clearing  rooms  while  wearing? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Does  the  system  prevent 

your  ability  to  move  with  stealth,  move  quietly,  move 

quickly? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Do  you  have  any  issues  with  donning  the  system? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 
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Do  you  have  any  issue  with  doffing  the  systems? 


prevents 

big  issues, 

noticeable 

small  issues  no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Do  you  have 

any  issues  with  the  sensitivity  of  the  mouse? 

prevents 

big  issues, 

noticeable 

small  issues  no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

What  effect  will  battery 

failure  have  on 

the  system? 

Catastrophic 

Major 

Moderate 

Minor  None 

How  does  the  system  compare  to  your  normal  equipment  with  respect  to  your 
units  lethality? 

much  worse  the  same  slightly  much  better 

better 

Compared  to  your  normal  equipment,  the  amount  of  time  it  takes  to  receive 
reports  from  your  units  members? 

much  worse  the  same  slightly  much  better 

better 

Compared  to  your  normal  equipment,  the  amount  of  time  it  takes  to  process 
reports  from  your  units  members? 

much  worse  the  same  slightly  much  better 

better 

Compared  to  your  normal  equipment,  the  amount  of  time  it  takes  to  pass  reports 
from  your  units  members? 

much  worse  the  same  slightly  much  better 

better 

Compared  to  your  normal  equipment,  what  is  the  impact  the  system  had  on  the 
accuracy  of  the  reports  processed? 

much  worse  the  same  slightly  much  better 

better 
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Compared  to  your  normal  equipment,  what  is  the  impact  the  system  had  on  the 
accuracy  of  the  reports  received? 

much  worse  the  same  slightly  much  better 

better 

Compared  to  your  normal  equipment,  what  is  the  impact  the  system  had  on  the 
accuracy  of  the  reports  passed  on? 

much  worse  the  same  slightly  much  better 

better 

Rate  the  ease/difficulty  to  integrate  the  system  with  existing  systems 

impossible  fairly  neutral  fairly  easy  no  problems 

difficult  at  all 

How  difficult  was  it  to  learn  to  put  on  the  system? 

very  moderately  neutral  moderately  very  easy 

difficult  difficult  easy 

How  does  the  system  compare  to  your  normal  equipment  with  respect  to  your 
ability  to  your  ability  to  request  and  adjust  fire  from  a  joint  source? 

much  worse  the  same  slightly  much  better 

better 

To  what  extent  should  users  be  restricted  from  system  use  due  to  physical  factors? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  does  a  fully  extended  antenna  give  away  system  position? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

How  many  times  did  you  change  the  batteries? 


To  what  extent  was  the  antenna  effective  in  transmitting  signals? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 


122 


Was  GPS  reception  steady  or  intermittent? 

Intermittent  Steady 

To  what  extent  does  software  glitches  require  system  reset? 


To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  does  the  system  require  a  backlight  to  make  screen  visible? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 


How  difficult  was  it  to  learn  to  assemble  the  system? 

very  moderately  neutral  moderately  very  easy 

difficult  difficult  easy 

The  system  improves  integration  of  other  systems 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

To  what  extent  does  the  system  require  daylight  conditions? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 


The  use  of  the  system  increased  understanding  of  the  mission 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Compared  to  normal  equipment  the  system  improves  the  units  overall  force 
effectiveness 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Compared  to  normal  equipment  the  system  improves  mobility 

strongly  disagree  neutral  agree  strongly 

disagree  agree 
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Compared  to  normal  equipment,  the  system  improves  the  mission  pace 


strongly 

disagree 


disagree 


neutral 


agree 


The  system  helps  the  unit  perform  its  mission  better 

strongly  disagree  neutral  agree 

disagree 

The  hindered  the  units  performance 

disagree  neutral 


strongly 

disagree 


agree 


strongly 

agree 


strongly 

agree 


strongly 

agree 


How  does  the  system  compare  to  your  normal  equipment  with  respect  to  your 
ability  to  coordinate  fires  and  movement  with  adjacent  units? 

much  worse  the  same  slightly  much  better 

better 


The  system  improves  accuracy  of  target  location  for  indirect  fire 

disagree  neutral  agree 


strongly 

disagree 


strongly 

agree 


How  does  the  system  compare  to  your  normal  equipment  with  respect  to  your 
ability  to  direct  the  fires  and  movement  of  subordinates? 

much  worse  the  same  slightly  much  better 

better 


The  use  of  the  system  improves  communication  between  members 

disagree  neutral  agree 


strongly 

disagree 


strongly 

agree 


The  use  of  the  system  improves  communication  between  units 

disagree  neutral  agree 


strongly 

disagree 


strongly 

agree 


The  system  decreases  the  length  of  time  it  takes  to  receive  complete  orders  with 
overlays 

strongly  disagree  neutral  agree  strongly 

disagree  agree 
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The  system  decreases  the  length  of  time  it  takes  to  understand  complete  orders 


with  overlays 

strongly  disagree 

neutral 

agree 

strongly 

disagree 

agree 

Compared  to  your  normal  equipment,  what  is  the  impact  of  the  new  system  on 
the  execution  of  leader  tasks? 

much  worse  the  same  slightly  much  better 

better 

How  does  the  system  compare  to  your  normal  equipment  with  respect  to  your 
ability  to  direct  employment  of  smoke? 

much  worse  the  same  slightly  much  better 

better 

How  does  the  system  compare  to  your  normal  equipment  with  respect  to  your 
ability  to  your  ability  to  conduct  engagements  with  precision  munitions? 


much  worse 

the  same 

slightly 

better 

much  better 

The  system  is  intuitively  designed 

completely  somewhat 

neutral 

somewhat 

completely 

disagree  disagree 

agree 

agree 

The  system  reduced  length  of  time  required  to  call  for  indirect  fire 

strongly  disagree 

neutral 

agree 

strongly 

disagree 

agree 

Is  it  easy  to  send  and  receive  short  messages  using  this  product? 

Very  Difficult 

Neither 

Easy 

Very  easy 

difficult 

difficult  nor 

easy 

Rate  the  ease  of  taking  the  item  off 

Very  Difficult 

Neither 

Easy 

Very  easy 

difficult 

difficult  nor 

easy 
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Is  the  interface  with  this  product  clear  and  understandable? 


To  no 

extent 

To  a  small 

extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

Are  the  characters  on  the  screen  easy  to  read? 

Very 

difficult 

Difficult 

Neither 
difficult  nor 

Easy 

Very  easy 

easy 

Does  interacting  with  this  product  require  a  lot  of  mental  effort? 


Yes  No 

Is  it  easy  to  assemble,  install,  and/or  setup  the  product? 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Can  you  regulate,  control,  and  operate  the  product  easily? 

No  Yes 

Is  the  presentation  of  system  information  sufficiently  clear  and  understandable? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Are  the  input  and  text  entry  methods  for  this  product  easy  and  usable? 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Is  it  easy  for  you  to  remember  how  to  perform  tasks  with  this  product? 

No  Yes 

Is  it  easy  to  use  the  phone  book  feature  of  this  product? 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 
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Is  it  easy  to  correct  mistakes  such  as  typos? 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 

Is  the  backlighting  feature  for  the  keyboard  and  screen  helpful? 


To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Are  the  command  names  meaningful? 

No  Yes 

Discovering  new  features  is  sufficiently  easy. 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Is  the  Web  interface  sufficiently  similar  to  those  of  other  products  you  have  used? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Did  you  have  access  to  properly  rated  electrical  components,  and  protective  for 
the  job? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Is  it  easy  to  navigate  between  hierarchical  menus,  pages,  and  screen? 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 

All  operations  can  be  carried  out  in  a  systematically  similar  way? 


strongly  disagree  neutral  agree  strongly 

disagree  agree 
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Is  it  easy  to  check  the  last  call? 


Very 

difficult 

Difficult 

Neither 
difficult  nor 
easy 

Easy 

Very  easy 

Are  exchange  and  transmission  of  data  between  this  product  and  other  products 
easy? 

Very 

difficult 

Difficult 

Neither 
difficult  nor 
easy 

Easy 

Very  easy 

In  general,  how  would  you  rate  the  ease  of  learning  the  user  interface? 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 

In  general,  how  would  you  rate  the  general  flexibility  the  user  interface? 


completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 

adequate 

adequate 

It  is  difficult  to  move  around  this  web  site 

Very 

Difficult 

Neither 

Easy 

Very  easy 

difficult  difficult  nor 


easy 

Is  it  easy  to  learn  to  operate  this  product? 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 

Is  using  this  product  sufficiently  easy? 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 

Is  it  sufficiently  easy  to  operate  keys  with  one  hand? 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 
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Is  it  relatively  easy  to  move  from  one  part  of  a  task  to  another? 


strongly  disagree  neutral 
disagree 

Did  the  radar  function  properly? 


agree 


strongly 

agree 


Are  the  operation  of  this  product  simple  and  uncomplicated? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

This  product  enables  the  quick,  effective,  and  economical  performance  of  tasks? 


strongly  disagree  neutral 
disagree 


agree 


strongly 

agree 


It  is  easy  to  access  the  information  that  you  need  from  the  product? 


strongly  disagree  neutral 
disagree 


agree 


strongly 

agree 


The  organization  of  information  on  the  product  screen  is  clear? 


strongly  disagree  neutral 
disagree 


agree 


strongly 

agree 


Does  product  have  all  the  functions  and  capabilities  you  expect  it  to  have? 


strongly  disagree  neutral 
disagree 


agree 


strongly 

agree 


Are  the  color  coding  and  data  display  compatible  with  familiar  conventions? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 


Is  the  documentation  and  manual  for  this  product  sufficiently  informative? 


completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 


129 


Have  the  user  needs  regarding  this  product  been  sufficiently  taken  into 
consideration? 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

The  response  time  and  information  display  fast  enough. 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Are  the  design  of  the  graphic  symbols,  icons  and  labels  on  the  icons  sufficiently 
relevant? 

completely  somewhat  neutral  somewhat  completely 

irrelevant  irrelevant  relevant  relevant 


Does  the  product  provide  index  of  commands? 

No 

Does  the  product  provide  index  of  data? 

No 

Are  data  items  kept  short? 

No 


Yes 


Yes 


Yes 


To  what  extent  are  the  letter  codes  for  the  menu  selection  designed  carefully? 


To  no 
extent 


To  a  small 
extent 


To  a 

moderate 

extent 


To  a  great 
extent 


To  a  very 
great  extent 


The  commands  have  distinctive  meanings. 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Is  the  HELP  information  given  by  this  product  useful? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

The  HOME  and  MENU  buttons  sufficiently  easy  to  locate  for  all  operations. 

disagree  neutral  agree 


strongly 

disagree 


strongly 

agree 
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Does  carrying  this  product  effect  your  performance? 


prevents  big  issues,  noticeable  small  issues  no  issues 

from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 

To  what  extent  has  the  product  at  some  time  stopped  unexpectedly? 

To  a  very  To  a  great  To  a  To  a  small  To  no 

great  extent  extent  moderate  extent 

extent 

Is  the  amount  of  information  displayed  on  the  screen  adequate? 


completely  somewhat  neutral 
inadequate  inadequate 


somewhat  completely 

adequate  adequate 


Is  the  way  product  works  overall  consistent? 

strongly  disagree  neutral  agree 

disagree 


strongly 

agree 


Does  the  product  allow  the  user  to  access  applications  and  data  with  sufficiently 
few  keystrokes? 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Is  the  data  display  sufficiently  consistent? 

prevents  big  issues,  noticeable  small  issues  no  issues 

from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 

Does  the  product  support  the  operation  of  all  the  tasks  in  a  way  that  you  find 
useful? 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Is  the  product  reliable,  dependable,  and  trustworthy? 

strongly  disagree  neutral  agree  strongly 

disagree  agree 
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The  highlighting  on  the  screen  helpful. 


strongly 

disagree 

neutral 

agree 

strongly 

disagree 

agree 

Does  the  brightness  of  the  screen  make  the  product  unusable? 

prevents 

big  issues, 

noticeable 

small  issues 

no  issues 

from  doing 

hard  to  do 

issue, 

but  easy  to 

your job 

my  job 

affects  my 
job 

live  with 

Are  the  messages  aimed  at  preventing  you  from  making  mistakes  adequate? 

completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 

adequate 

adequate 

Are  the  error  messages  effective  in  assisting  you  to  fix  problems? 

To  no 

To  a  small 

To  a 

To  a  great 

To  a  very 

extent 

extent 

moderate 

extent 

extent 

great  extent 

Is  it  easy  to  take  corrective  actions  once 

an  error  has  been  recognized? 

Very 

Difficult 

Neither 

Easy 

Very  easy 

difficult 

difficult  nor 

easy 

Is  feedback  on  the  completion  of  tasks  clear? 

To  no 

To  a  small 

To  a 

To  a  great 

To  a  very 

extent 

extent 

moderate 

extent 

extent 

great  extent 

Does  the  product  give  all  the  necessary  information  for  you  to  use  it  in  a  proper 

manner? 

To  no 

To  a  small 

To  a 

To  a  great 

To  a  very 

extent 

extent 

moderate 

extent 

extent 

great  extent 

To  what  extent  is  the  bolding  of  commands  or  other  signals  helpful? 

To  no 

To  a  small 

To  a 

To  a  great 

To  a  very 

extent 

extent 

moderate 

extent 

extent 

great  extent 
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To  what  extent  does  the  HELP  function  define  aspects  of  the  product  adequately? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Is  the  organization  of  the  menus  sufficiently  logical? 

completely  somewhat  neutral  somewhat  completely 

illogical  illogical  logical  logical 

Does  the  color  of  the  product  make  it  unusable? 

prevents  big  issues,  noticeable  small  issues  no  issues 

from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 

Can  you  personalize  warning  signals  with  this  product? 

No  Yes 

Are  pictures  on  the  screen  of  satisfactory  quality  and  size? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Is  the  number  of  colors  available  adequate? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Are  the  components  of  the  product  are  well-matched  or  harmonious? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Do  you  feel  excited  when  using  this  product? 

No  Yes 

Would  you  miss  this  product  if  you  no  longer  had  it? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 
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Are  you/would  you  be  happy  to  have  this  product? 

No  Yes 

Is  it  easy  to  change  the  ringer  signal? 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Is  this  product's  size  convenient  for  transportation  and  storage? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

List  the  3  best  attributes/features  of  the  system: 


How  long  have  you  been  operating  the  system? 


How  long  does  it  take  to  assemble  the  system? 


How  long  does  it  take  to  disassemble  the  system? 


Rate  the  normal  day-to-day  performance  of  the  system. 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Rate  the  ability  to  transmit/receive  on  uneven  or  hilly  terrain. 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Did  the  system  interfere  with  the  operation  of  the  weapon  to  which  it  was 

mounted? 

prevents  big  issues,  noticeable  small  issues  no  issues 

from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 
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Did  the  placement  of  the  antenna  interfere  with  the  operation  of  the  roof- 
mounted  crew  served  weapon? 

prevents  big  issues,  noticeable  small  issues  no  issues 

from  doing  hard  to  do  issue,  but  easy  to 

your  job  my  job  affects  my  live  with 

job 

Rate  the  clarity  of  the  PA  speaker? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

List  the  3  least  liked  attributes/features  of  the  system: 

Is  it  easy  to  check  missed  calls? 


Very 

difficult 

Difficult 

Neither  Easy 

difficult  nor 

Very  easy 

easy 

Provide  any  other  recommendations  for  improving  or  using  this  system: 

Were  there  any  components  that  did  not  perform  well? 

Were  there  any  components  that  had  reliability  problems? 

Were  there  any  components  that  had  maintenance  problems? 

Rate  the  mounting  of  the  Antenna  (able  to  withstand  high  winds/bursts  of  speed)? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Are  the  non-lethal  aspects  of  the  kit  appropriate?  Why? 

Was  Video  interference  a  factor? 

Yes  No 
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On  what  type  of  mission  was  the  system  employed?  (i.e.  Counter  IED 
surveillance,  FOB  surveillance,  Detention  facility  surveillance)  Briefly 


Is  the  tool  kit  issued  with  the  system  sufficient?  If  not,  what  would  you  add 

No  Yes 

How  many  hours  (at  a  time)  do  you  typically  use  this  system?  (Please  circle  only 
one) 

Rate  your  visual  performance  while  using  the  product. 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Rate  your  Visual  Depth  Perception  while  using  the  product. 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Was  it  the  reaction  you  anticipated? 


Rate  the  Visual  Image  Clarity  while  using  the  product. 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

How  did  people  respond/react  to  the  use  of  this  item? 


At  what  range  was  this  system  employed? 


Rate  the  image  quality  of  the  photographs? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Rate  the  information  contained  on  the  photograph? 


completely  somewhat  neutral 
inadequate  inadequate 


somewhat  completely 

adequate  adequate 
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Rate  the  portability  of  this  item  (ease  of  moving)? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

Rate  the  image  quality  of  the  illuminated  sign? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

List  any  of  the  components  that  you  normally  did  not  use,  and  specify  why  you 

did  not  use  it  (them). 

Rate  your  Visual  Field  of  View  while  using  the  product. 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

In  general,  how  would  you  rate  the  visual  design  of  the  user  interface? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

What  are  any  recommendations  for  improving  or  using  the  system? 

In  general,  how  would  you  rate  the  error  messages  in  the  user  interface? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

In  general,  how  would  you  rate  the  task  layout/task  flow  of  the  user  interface? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

In  general,  how  would  you  rate  the  layout  of  the  user  interface? 

completely  somewhat  neutral  somewhat  completely 

inadequate  inadequate  adequate  adequate 

How  long  did  it  take  to  set  up? 
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Provide  any  recommendations  for  improving  or  using  this  system: 


Soldiers  can  effectively  complete  their  mission  using  this  equipment. 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

In  general,  how  would  you  rate  the  ease  of  use  of  the  user  interface? 

Very  Difficult  Neither  Easy  Very  easy 

difficult  difficult  nor 

easy 

Did  this  system  perform  as  expected? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Overall,  I  am  satisfied  with  how  easy  it  is  to  use  this  system. 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Specify  any  problems  identified  in  operating  the  system. 


To  what  extent  did  the  system  perform  as  expected? 


To  no  To  a  small 

extent  extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

LEADERSHIP 

To  what  extent  do  the  senior  officers  understand  the  scope  of  the  problem? 


To  no 

extent 

To  a  small 

extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

To  what  extent  does  the 

issue  effect  CC 

or  JTF’s  conduct  of  Joint  operations? 

To  a  very 
great  extent 

To  a  great 
extent 

To  a 

moderate 

extent 

To  a  small 

extent 

To  no 
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To  what  extent  has  senior  leadership  identified  interservice/agency  cultural 
drivers  and  barriers  which  hinder  issue  resolution? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  does  command  have  resources  at  its  disposal  to  correct  the  issue? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  has  the  command  properly  assessed  the  level  of  criticality,  threat, 
urgency,  risk,  etc.  of  the  operational  results  as  it  relates  to  the  issue? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  is  the  senior  leadership  aware  of  the  drivers  and  barriers  to 
resolving  the  issue  within  her/his  own  organization? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  is  leadership  being  trained  on  effective  change  management 
principles? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 


MANPOWER 

To  what  extent  does  the  system  increase  human  performance? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Additional  personnel  are  needed  to  support  the  maintenance  of  the  system 

Strongly  Agree  Neutral  Disagree  Strongly 

agree  Disagree 

Will  the  system  increase  manpower  requirements? 

Yes  No 
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Additional  personnel  are  needed  for  the  logistical  support  of  the  system 

Strongly  Agree  Neutral  Disagree  Strongly 

agree  Disagree 

The  capabilities  of  the  system  will  result  in  the  elimination  of  the  currently 
authorized  personnel 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

The  system  will  required  adding  personnel  with  MOSs  not  currently  in  the  unit 

Strongly  Agree  Neutral  Disagree  Strongly 

agree  Disagree 

To  what  extent  are  total  manpower  requirements  within  Unit  constraints? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Compared  to  your  normal  equipment,  what  is  the  impact  of  the  new  system  on 
the  execution  of  squad  and  platoon  collective  tasks? 

much  worse  the  same  slightly  much  better 

better 


MATERIEL 

To  what  extent  does  the  inability  or  decreased  ability  to 

cooperate/coordinate/communicate  with  external  organizations  impact  the  issue? 

To  a  very  To  a  great  To  a  To  a  small  To  no 

great  extent  extent  moderate  extent 

extent 

To  what  extent  does  inadequate  or  outdated  systems  and  equipment  impact  the 
issue? 

To  a  very  To  a  great  To  a  To  a  small  To  no 

great  extent  extent  moderate  extent 

extent 

Is  maintenance  support  available  in  theater? 

No  Yes 

What  current  systems  are  in  the  Family-of-Systems  where  the  problem  is 
occurring? 
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To  what  extent  should  the  Army  investigate  further  use  of  this  system  and 
similar  systems? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Is  online  Tech  support  available? 

No  Yes 

If  the  system  needed  parts  then  were  the  parts  readily  available? 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

The  capabilities  of  the  system  will  require  the  unit  to  get  additional  equipment  to 
complete  its  mission 

Strongly  Agree  Neutral  Disagree  Strongly 

agree  Disagree 

The  capabilities  of  the  system  will  make  some  equipment  in  my  unit  no  longer 
necessary 

Strongly  Agree  Neutral  Disagree  Strongly 

agree  Disagree 

To  what  extent  can  performance  increases  be  achieved  without  the  development 
of  a  new  system? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

In  your  opinion,  should  the  Army  investigate  further  use  of  this  system  and 
similar  system?  Explain: 

No  Yes 

What  functionality  would  a  new  system  provide  that  currently  does  not  exist? 


If  contractor  support  was  requested  how  long  did  it  take  to  get  the  system  fixed? 


When  the  item  was  deadlined  did  the  unit  fix  the  system  or  was  contractor 
support  requested?  Unit  Contractor 
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Was  the  item  ever  deadlined?  Y  N  If  yes,  why  and  for  how  long? 

Yes  No 

Who  would  be  the  primary  and  secondary  users  of  the  proposed  systems  or 
equipment? 

What  increases  in  operational  performance  are  needed  to  resolve  the  issue? 


NONE 

Use  space  below  to  write  any  additional  comments  about  the  system. 


ORGANIZATION 

To  what  extent  does  the  organization  have  the  funding  to  address  the  issue? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

What  duty  position  where  you  filling  when  you  used  this  system? 


Normally,  what  duty  position  were  you  filling  when  using  this  system? 


The  capabilities  provided  by  the  system  required  changes  in  the  platoon's 


organizational  structure 

Strongly  Agree 

Neutral 

Disagree 

Strongly 

agree 

Disagree 

Personnel 

To  what  extent  is  the  issue  impacted  by  the  inability  or  decreased  ability  to  place 
qualified  and  trained  personnel  in  occupational  specialties? 

To  a  very  To  a  great 

To  a 

To  a  small 

To  no 

great  extent  extent 

moderate 

extent 

extent 
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To  what  extent  do  new  training  programs  need  to  be  developed  for  newly 
recruited  personnel? 

To  a  very  To  a  great  To  a  To  a  small  To  no 

great  extent  extent  moderate  extent 

extent 

If  issue  resolution  is  likely  to  involve  new  material,  systems,  or  equipment,  are 
different  occupational  specialty  codes  needed  to  properly  staff  new  systems? 

To  a  very  To  a  great  To  a  To  a  small  To  no 

great  extent  extent  moderate  extent 

extent 


The  system  is  too  complex  for  most  soldiers  to  master 

Strongly  Agree  Neutral  Disagree 

agree 


Strongly 

Disagree 


The  system's  tasks  are  the  same  for  all  skill  levels  within  each  MOS 

disagree  neutral  agree 


strongly 

disagree 


strongly 

agree 


To  what  extent  will  the  system  likely  increase  human  performance? 


To  no 
extent 


To  a  small 
extent 


To  a 

moderate 

extent 


To  a  great 
extent 


To  a 

moderate 

extent 


Survivability 

Did  the  item  encounter  any  IEDs? 


Rate  the  effectiveness  of  the  suppressor  flash 

Very  Ineffective  Neither  Effective  Very 

ineffective  ineffective  effective 

nor  effective 

The  use  of  the  system  improves  the  units  ability  to  locate  mines  and  booby  trap 

agree 


strongly  disagree  neutral 
disagree 


strongly 

agree 


At  what  speed  did  you  encounter  the  IEDs? 
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To  what  extent  does  the  system  increase  mission  performance? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  does  the  system  reduce  situational  awareness? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  does  the  system  increase  your  personal  protection? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  does  the  backlight  of  the  system  give  away  position? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

The  use  of  the  system  improves  the  units  ability  to  avoid  enemy  fire 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Compared  to  normal  equipment  the  system  improves  the  unit's  survivability 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

Rate  the  effectiveness  of  the  suppressor  in  reducing  the  acoustic  signature  of  the 
weapon. 

Very  Ineffective  Neither  Effective  Very 

ineffective  ineffective  effective 

nor  effective 


TRAINING 

To  what  extent  can  Government  and  contractor  personnel  be  trained  to  complete 
all  system-related  tasks? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 
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How  effective  is  the  delivery  of  the  training? 


Very  Ineffective 

Neither 

Effective 

Very 

ineffective 

effective 

effective 

nor 

To  what  extent  does  current  training  exist  to  address  the  issue? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  is  the  current  issue  caused  by  a  complete  lack  of  or  inadequate 
training? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

What  training  methods  and  technologies  will  be  used? 


To  what  extent  has  the  personnel  flow  through  the  personnel  pipelline  been 
evaluated? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

Overall,  how  frequently  will  sustainment  or  refresher  training  be  needed? 


To  a  very 
great  extent 

To  a  great 
extent 

To  a 

moderate 

extent 

To  a  small 
extent 

To  no 

To  what  extent  have  training  objectives 

been  specified  in  detail? 

To  no 

extent 

To  a  small 

extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

To  what  extent  do  affected 

personnel  have  access  to  training? 

To  no 
extent 

To  a  small 
extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 
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To  what  extent  have  the  training  objectives  been  specified  in  sufficient  detail? 


To  no 

extent 

To  a  small 

extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

To  what 

extent  is  computer  literacy  required? 

To  no 
extent 

To  a  small 
extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

To  what 

extent  does  the  training  address  the  original 

10-Liner  Requirement? 

To  no 

extent 

To  a  small 

extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

The  Training  System  is  designed  to  improve  skillsets  in  which  of  the  HSI 
domains? 


To  what  extent  is  the  training  given  to  the  troops  valid  for  the  task? 


To  no 

extent 

To  a  small 

extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

How  effectively  can  the  reliability  of  the  training  be  measured? 

Very 

ineffective 

Ineffective 

Neither 

effective 

Effective 

Very 

effective 

nor 

To  what  extent  will  the  user  have  an  opportunity  to  practice  with  the  equipment 
prior  to  using  it  in  a  true  operational  setting? 


To  no 
extent 

To  a  small 
extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

To  what  extent  does  the 

training  system 

change  personnel  requirements? 

To  a  very 
great  extent 

To  a  great 
extent 

To  a 

moderate 

extent 

To  a  small 
extent 

To  no 

146 


What  effect  does  the  training  have  on  decreasing  manpower  requirements? 


Very  Ineffective 

Neither 

Effective 

Very 

ineffective 

effective 

effective 

nor 

Will  training  be  at  the  individual,  crew/team,  or  unit  level? 


Adequate  training  materials  or  support  is  available. 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

To  what  extent  have  the  system's  critical  tasks  been  identified  for  support 
personnel? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

If  it  is  not  possible  to  simplify  the  task,  to  what  extent  is  more  training  required? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  is  training  properly  funded? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

How  effectively  will  the  training  system  enable  soldiers  to  complete  currently 
assigned  duties? 

Very  Ineffective  Neither  Effective  Very 

ineffective  ineffective  effective 

nor  effective 

To  what  extent  does  lack  of  proper  environment  controls  impact  the  issue? 

To  a  very  To  a  great  To  a  To  a  small  To  no 

great  extent  extent  moderate  extent 

extent 
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To  what  extent  can  government  and  contractor  personnel  be  trained  on  all 
system-related  tasks  within  the  time  allotted? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

How  difficult  was  it  to  learn  to  maintain  the  system? 

very  moderately  neutral  moderately  very  easy 

difficult  difficult  easy 

To  what  extent  are  training  results  being  measured  and  monitored? 


To  no 

extent 

To  a  small 

extent 

To  a 

moderate 

extent 

To  a  great 
extent 

To  a  very 
great  extent 

Rate  the  training  that  you  received  on  this  system. 

Very 

ineffective 

Ineffective 

Neither 

effective 

Effective 

Very 

effective 

nor 

To  what  extent  is  the  issue  caused  by  a  lack  of  competency  or  proficiency  on 
existing  systems  and  equipment? 

To  a  very  To  a  great  To  a  To  a  small  To  no 

great  extent  extent  moderate  extent 

extent 

What  percentage  of  the  target  population  is  capable  of  understanding  the 
training  required  to  properly  operate  the  new  system? 

up  to  20%  21%  -  41%  -  61%  -  81%  -  100% 


Were  the  installation  procedures  accurately  portrayed  in  the  Training  Support 
Package? 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

How  difficult  was  it  to  learn  to  remove  the  system? 


very  moderately  neutral  moderately  very  easy 

difficult  difficult  easy 
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New  arrivals  to  the  unit  will  be  able  to  use  the  system  with  only  OJT 

strongly  disagree  neutral  agree  strongly 

disagree  agree 

To  what  extent  is  the  training  properly  staffed? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  is  the  command/management  supporting  and/or  enforcing  the 
training  effort? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

To  what  extent  have  the  system's  critical  tasks  been  identified  for  maintainers? 

To  no  To  a  small  To  a  To  a  great  To  a  very 

extent  extent  moderate  extent  great  extent 

extent 

How  difficult  was  it  to  learn  to  operate  the  system? 

very  moderately  neutral  moderately  very  easy 

difficult  difficult  easy 
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APPENDIX  B.  REF  SYSTEM  ASSESSMENT  FORM 


System  Assessment  Form  -  System/Item  Name: _ Date:  _ 

Location: _  Unit:  _ 

Rate  the  following  from  1  to  10  (circle  your  answer). 

Terrible  Poor  Adequate  Good  Excellent 

Performance:  Does  the  system/item  perfonn  as  expected? 

123456789  10 

Effectiveness:  Does  the  system/item  meet  requirements? 

123456789  10 

Survivability:  Is  the  system/item  adequately  built  for  field  use? 

12345678  9  10 

Training:  Are  adequate  training  materials/support  available? 

12345678  9  10 

Reliability:  Does  the  system/item  hold  up  under  continual  use? 

1234567  89  10 

Maintainability:  Is  the  system/item  easily  cleaned  &  fixed? 

1234567  8  9  10 

Supportability:  Are  parts/procedures/support  readily  available? 

1234567  8  9  10 

Operation:  Is  the  system/item  easy  to  operate? 

1234567  89  10 

Interoperation:  How  well  does  the  system  operate  with  other  systems,  if  required? 
1234567  89  10 

Employment:  If  this  system  was  fielded,  would  you  use  it?  Yes  No 
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How  would  you  use  it? 

PROS:  List  and  explain  at  least  3  things  you  like  about  the  system/item. 

CONS:  List  and  explain  at  least  3  things  you  do  not  like  about  the  system/item. 
Item  Improvement:  What  suggestions  can  you  provide  to  improve  the  system/item. 


152 


APPENDIX  C.  ABRAHAM  ANALYSIS  REPORT 


1 .  The  following  report  is  based  on  a  review  of  the  individual  HSI  domains 
and  the  responses  to  the  questions  generated  by  this  tool.  The  goal  of  this  report 
is  to  identify  the  question  responses  that  imply  critical,  major  or  moderate  risks  to 
acceptable  perfonnance  of  the  human  and  system  in  the  operational  environment. 
This  assessment  relies  upon  the  honest  and  forthright  responses  of  the  user,  and 
any  efforts  to  address  these  HSI  issues  will  require  that  leadership  verify  the 
deficiencies  exist. 

2.  The  overall  score  for  this  piece  of  equipment  is  based  on  your  responses  to 
the  questions  and  the  importance  rating  given  to  each  of  the  questions. 

3.  The  following  questions  indicate  that  there  may  be  issues  with  the  system 
that  would  cause  unnecessary  performance  decrement  that  could  adversely  impact 
Soldier  safety,  survivability,  and  lethality.  Specific  issues  are  identified  as 
follows: 

CRITICAL  ISSUES: 

Question: 

What  percentage  of  the  target  population  is  capable  of  understanding  the  training 
required  to  properly  operate  the  new  system? 

Your  Response:  up  to  20% 

Question  Score:  3  Maximum  Possible  15 

Additional  Comments: 

The  training  is  hard  to  understand.  Nobody  has  scored  acceptably  on  the  exit 
exam. 

Major  Issues: 

Question: 

To  what  extent  is  the  current  issue  caused  by  a  complete  lack  of  or  inadequate 
training? 

Your  Response:  To  a  small  extent 
Question  Score:  6  Maximum  Possible:  15 

Additional  Comments: 
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Question: 

Overall,  how  frequently  will  sustainment  or  refresher  training  be  needed? 

Your  Response:  To  a  great  extent 
Question  Score:  2  Maximum  Possible:  10 

Additional  Comments: 

Question: 

To  what  extent  can  Government  and  contractor  personnel  be  trained  to  complete 
all  system-related  tasks? 

Your  Response: 

To  a  small  extent 

Question  Score:  4  Maximum  Possible:  10 

Additional  Comments: 

Question: 

To  what  extent  have  the  training  objectives  been  specified  in  sufficient  detail? 
Your  Response:  To  a  small  extent 

Question  Score:  4  Maximum  Possible  10 

Additional  Comments: 


MODERATE  ISSUES: 

Question: 

To  what  extent  have  training  objectives  been  specified  in  detail? 

Your  Response:  To  a  moderate  extent 

Question  Score:  6  Maximum  Possible  10 

Additional  Comments: 

Question: 

To  what  extent  will  the  user  have  an  opportunity  to  practice  with  the  equipment 
prior  to  using  it  in  a  true  operational  setting? 

Your  Response:  To  a  moderate  extent 

Question  Score:  3  Maximum  Possible:  15 

Additional  Comments: 


Overall  Score:  28  Out  of  a  maximum  of:  85 
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